You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@nifi.apache.org by "wanglei2@geekplus.com.cn" <wa...@geekplus.com.cn> on 2019/10/28 08:14:06 UTC
How to deploy NiFi processors change to multiple NiFi instances?
We have many standalone NiFi instances and all running the same NiFi Flow.
If the flow changes, how to deploy the change to all NiFi instances automatically?
Thanks,
Lei
wanglei2@geekplus.com.cn
RE: How to deploy NiFi processors change to multiple NiFi instances?
Posted by "Woodcock, Michael W., M.B.A." <wo...@mayo.edu>.
The approach we use for disaster recovery might be a start.
* nifi (user) has a cron job that runs BASH script on PROD server node which copies the flow.xml.gz file to be used on another cluster to a temp folder.
* The script decrypts the current passwords/properties in flow.xml.gz and encrypts with new BACKUP's password which re-encrypts sensitive properties.
* /var/lib/ambari-agent/tmp/nifi-toolkit-1.5.0.3.1.0.0-564/bin/encrypt-config.sh -b /usr/hdf/current/nifi/conf/bootstrap.conf -f /apps/var/lib/nifi/conf/flow.xml.gz -g ~/flow.xml.gz -n /usr/hdf/current/nifi/conf/nifi.properties -o /usr/hdf/current/nifi/conf/backup/nifi.properties -s $#$#$#$#$#$#$#$ -x
* Script also unzips flow.xml.gz, makes modifications by calling python script that changes a few labels and parameters using xml.etree.
* The flow.xml.gz is zipped and copied to /tmp with chmod changes
* Another cron copies the flow.xml.gz to the BACKUP servers offsite.
* Cron on BACKUP servers runs script that stops NiFi, copies new flow.xml.gz overwriting existing one. Restarts NiFi with all processors stopped. Otherwise system starts with flows in last known state in PROD.
Caveats are:
NiFi versions and configurations the same
Users should be the same
Should point to same NiFi Registry, if used
If using Ranger, run a script exporting configuration, renaming it and sending it to BACKUP's Ranger
User authorization and authentication mechanisms the same.
Note that the password for encrypting flow for the BACKUP server is exposed in script ($#$#$#$#$#$#$#$).
I don't remember what else...
Mike
From: Andy LoPresto [mailto:alopresto@apache.org]
Sent: Monday, October 28, 2019 12:09 PM
To: users@nifi.apache.org
Subject: Re: How to deploy NiFi processors change to multiple NiFi instances?
I would generally want a human intervention step in this process, but if you want this to be fully-automated, you can script that using the Event Hooks [1] feature. As Edward noted, the NiFi Toolkit or NiPyAPI could be invoked from here to execute whatever steps you want taken.
[1] https://nifi.apache.org/docs/nifi-registry-docs/html/administration-guide.html#event-hooks
Andy LoPresto
alopresto@apache.org<ma...@apache.org>
alopresto.apache@gmail.com
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
On Oct 28, 2019, at 1:38 AM, Edward Armes <ed...@gmail.com>> wrote:
Hi Lei,
As far as I'm aware there currently isn't an out of the box method to do this
However I would look at using a combination of the nifi registry and tools in nifi toolkit to roll your own.
Also, there have been multiple questions to both this and the dev mailing list, asking what could be done. I would also suggest searching the list archives and see what others did and didn't do as well.
Edward
On Mon, 28 Oct 2019, 08:14 wanglei2@geekplus.com.cn<ma...@geekplus.com.cn>, <wa...@geekplus.com.cn>> wrote:
We have many standalone NiFi instances and all running the same NiFi Flow.
If the flow changes, how to deploy the change to all NiFi instances automatically?
Thanks,
Lei
________________________________
wanglei2@geekplus.com.cn<ma...@geekplus.com.cn>
Re: How to deploy NiFi processors change to multiple NiFi instances?
Posted by Andy LoPresto <al...@apache.org>.
I would generally want a human intervention step in this process, but if you want this to be fully-automated, you can script that using the Event Hooks [1] feature. As Edward noted, the NiFi Toolkit or NiPyAPI could be invoked from here to execute whatever steps you want taken.
[1] https://nifi.apache.org/docs/nifi-registry-docs/html/administration-guide.html#event-hooks
Andy LoPresto
alopresto@apache.org
alopresto.apache@gmail.com
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
> On Oct 28, 2019, at 1:38 AM, Edward Armes <ed...@gmail.com> wrote:
>
> Hi Lei,
>
> As far as I'm aware there currently isn't an out of the box method to do this
> However I would look at using a combination of the nifi registry and tools in nifi toolkit to roll your own.
>
> Also, there have been multiple questions to both this and the dev mailing list, asking what could be done. I would also suggest searching the list archives and see what others did and didn't do as well.
>
> Edward
>
> On Mon, 28 Oct 2019, 08:14 wanglei2@geekplus.com.cn <ma...@geekplus.com.cn>, <wanglei2@geekplus.com.cn <ma...@geekplus.com.cn>> wrote:
>
> We have many standalone NiFi instances and all running the same NiFi Flow.
> If the flow changes, how to deploy the change to all NiFi instances automatically?
>
> Thanks,
> Lei
>
>
> wanglei2@geekplus.com.cn <ma...@geekplus.com.cn>
Re: How to deploy NiFi processors change to multiple NiFi instances?
Posted by Edward Armes <ed...@gmail.com>.
Hi Lei,
As far as I'm aware there currently isn't an out of the box method to do
this
However I would look at using a combination of the nifi registry and tools
in nifi toolkit to roll your own.
Also, there have been multiple questions to both this and the dev mailing
list, asking what could be done. I would also suggest searching the list
archives and see what others did and didn't do as well.
Edward
On Mon, 28 Oct 2019, 08:14 wanglei2@geekplus.com.cn, <
wanglei2@geekplus.com.cn> wrote:
>
> We have many standalone NiFi instances and all running the same NiFi
> Flow.
> If the flow changes, how to deploy the change to all NiFi instances
> automatically?
>
> Thanks,
> Lei
>
>
> ------------------------------
> wanglei2@geekplus.com.cn
>