You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@nifi.apache.org by wildo <wi...@moserit.com> on 2017/11/08 04:58:05 UTC
Custom properties file for environmental promotion
We have nearly wrapped up our testing with out Nifi scripts in dev, and are
now looking to push to QA. I found an article about creating a custom
properties file in order to specify each of your environmental specific
variables, and then specifying that file in nifi.properties at
nifi.variable.registry.properties.
This will work fine omitting two cases I can think of.
1) We have a number of ListenTCP processors which require the "local network
interface" to be specified. I have read that Expression Language can access
system properties, but I haven't seen any example about how to use this. Can
anyone share how EL might be used to grab the local network interface for
each environment automatically?
2) We use Remote Process Groups with Site-to-Site for load balancing. In the
RPG, you have to specify an absolute url to the nodes in the remote site.
The RPG doesn't indicate that EL is acceptable in this field. Can anyone
chime in on the possibility of using EL to grab a property for the RPG url?
Thanks!
--
Sent from: http://apache-nifi-developer-list.39713.n7.nabble.com/
Re: Custom properties file for environmental promotion
Posted by wildo <wi...@moserit.com>.
Thanks Bryan.
I just wrapped this up and wanted to document for anyone in the future.
-Yes, updating the kerberos principal/keytab values via the xml template
worked just fine.
-Yes, updating the target url fields on the RPG worked just fine.
-Yes, Bryan was absolutely correct that you DO need to update the port ID in
the xml template to point to the proper ID in the *new* environment. The big
gotchya is that this is not only specified in the <remoteProcessGroup />
itself, but the ID is also specified as a <connection /> on the parent
process group that it lives in. If you miss this, then Nifi will throw an
error when you try to add the template to the canvas stating something like
"An ID to a connection was specified that does not exist." Sorry I don't
have explicit error message in front of me.
In summary: since we are limited to Nifi 1.2, this manual updating of the
XML worked fine. Just be mindful of the gotchya mentioned above.
--
Sent from: http://apache-nifi-developer-list.39713.n7.nabble.com/
Re: Custom properties file for environmental promotion
Posted by Bryan Bende <bb...@gmail.com>.
In general that approach should work, there were a few community
efforts that did something like this in the past [1][2].
For the RPG, you may need to substitute another value as well, because
I believe the template also contains the UUID of the ports it is
connected to, which will be different depending on the target URI, but
I don't remember exactly how this part works, maybe others can chime
in.
Thanks,
Bryan
[1] https://github.com/aperepel/nifi-api-deploy
[2] https://github.com/hermannpencole/nifi-config
On Mon, Nov 13, 2017 at 1:47 PM, wildo <wi...@moserit.com> wrote:
> If you guys would please entertain one more question:
>
> Unfortunately the site is currently limited to Nifi 1.2, and this
> effectively renders the custom properties file useless as it relates to the
> PutHDFS kerberos principal/keytab. I was thinking about this as I exported
> the template in order to put into source control.
>
> Is there any reason that I couldn't modify the xml template to replace these
> values before uploading the template into the new environment?
>
> <entry>
> <key>Kerberos Principal</key>
> <value>dev@mydomain.com</value>
> </entry>
>
> <entry>
> <key>Kerberos Keytab</key>
> <value>/my/path/to/keytabs/dev.keytab</value>
> </entry>
>
>
> Similarly, when it comes to the remote process group, is there any reason
> that I can't just modify these uris before uploading the template to the new
> environment?
>
> <targetUri>https://dev001.mydomain.com:1234/nifi</targetUri>
> <targetUris>https://dev001.mydomain.com:1234/nifi,https://dev002.mydomain.com:1234/nifi</targetUris>
>
>
> It would be quite simple to write a script that does a simple substitution
> on these values. But is there any reason that that wouldn't work? I imagine
> I'm hardly the first to think about editing the template xml itself...
>
>
>
> --
> Sent from: http://apache-nifi-developer-list.39713.n7.nabble.com/
Re: Custom properties file for environmental promotion
Posted by wildo <wi...@moserit.com>.
If you guys would please entertain one more question:
Unfortunately the site is currently limited to Nifi 1.2, and this
effectively renders the custom properties file useless as it relates to the
PutHDFS kerberos principal/keytab. I was thinking about this as I exported
the template in order to put into source control.
Is there any reason that I couldn't modify the xml template to replace these
values before uploading the template into the new environment?
<entry>
<key>Kerberos Principal</key>
<value>dev@mydomain.com</value>
</entry>
<entry>
<key>Kerberos Keytab</key>
<value>/my/path/to/keytabs/dev.keytab</value>
</entry>
Similarly, when it comes to the remote process group, is there any reason
that I can't just modify these uris before uploading the template to the new
environment?
<targetUri>https://dev001.mydomain.com:1234/nifi</targetUri>
<targetUris>https://dev001.mydomain.com:1234/nifi,https://dev002.mydomain.com:1234/nifi</targetUris>
It would be quite simple to write a script that does a simple substitution
on these values. But is there any reason that that wouldn't work? I imagine
I'm hardly the first to think about editing the template xml itself...
--
Sent from: http://apache-nifi-developer-list.39713.n7.nabble.com/
Re: Custom properties file for environmental promotion
Posted by wildo <wi...@moserit.com>.
Thanks Matt! I was just coming back to update. I found the 1.4 docs that
indicated that it was available there. We are currently using 1.2. I should
have researched better... Thanks again!
--
Sent from: http://apache-nifi-developer-list.39713.n7.nabble.com/
Re: Custom properties file for environmental promotion
Posted by Matt Burgess <ma...@apache.org>.
It looks like the HDFS ones were added in NiFi 1.4.0 [1] [2], and the
ListenTCP EL support for Port has an open Jira [3].
Regards,
Matt
[1] https://issues.apache.org/jira/browse/NIFI-2923
[2] https://issues.apache.org/jira/browse/NIFI-4010
[3] https://issues.apache.org/jira/browse/NIFI-4458
On Mon, Nov 13, 2017 at 11:08 AM, wildo <wi...@moserit.com> wrote:
> Oh no! I just realized that the Kerberos Principle and Keytab fields in
> PutHDFS do *not* support expression language. Oy. This is a nonstarter...
>
> Bryan- is there some way to submit an enhancement request for EL support on
> these fields? Same goes for the port number on ListenTCP- no EL support.
> Sorry- I'm not aware of the protocol for submitting such an enhancement.
>
>
>
> --
> Sent from: http://apache-nifi-developer-list.39713.n7.nabble.com/
Re: Custom properties file for environmental promotion
Posted by wildo <wi...@moserit.com>.
Oh no! I just realized that the Kerberos Principle and Keytab fields in
PutHDFS do *not* support expression language. Oy. This is a nonstarter...
Bryan- is there some way to submit an enhancement request for EL support on
these fields? Same goes for the port number on ListenTCP- no EL support.
Sorry- I'm not aware of the protocol for submitting such an enhancement.
--
Sent from: http://apache-nifi-developer-list.39713.n7.nabble.com/
Re: Custom properties file for environmental promotion
Posted by wildo <wi...@moserit.com>.
Excellent info, as always Bryan. Very much appreciated!
--
Sent from: http://apache-nifi-developer-list.39713.n7.nabble.com/
Re: Custom properties file for environmental promotion
Posted by Bryan Bende <bb...@gmail.com>.
Currently, there is the variable properties file which would require a
service restart and also would need to be on all nodes in a cluster.
The last release (1.4.0) added a more user-friendly variable registry
in the UI which you can access from the context palette for a given
process group, near where the controller services for a PG are
located.
When editing variables in the UI, it will detect components that
reference them and automatically restart them. This variable registry
will be tightly integrated with the experience of using the flow
registry.
Given all of the above, there still isn't anything you can currently
do for RPGs though... you will unfortunately have to recreate it in
the target environment until they become editable.
As far as when the registry will be released, there are no set
timelines for apache projects so it will be based when the community
believes it is mature enough to be released, and when someone
volunteers to be the release manager.
That being said, a lot of good work has been done already and it is
maturing quickly.
Thanks,
Bryan
On Wed, Nov 8, 2017 at 10:21 AM, wildo <wi...@moserit.com> wrote:
> Great info Bryan- thanks!
>
> Regarding my first question, I talked to our admins and we only have one NIC
> anyway. So there is no need for me to limit it, and thus I don't have a need
> to use EL to discover the NIC. So that's good.
>
> Regarding the registry stuff, I found this [1] document which looks
> FANTASTIC. But I'm not able to find when/if this stuff will be released. My
> understanding is that it is not yet released, and therefore I'm assuming
> that specifying a custom.properties file via the
> nifi.variable.registry.properties is still the preferred method.
> Additionally, this will mean that:
> 1) Changes to this file require a service restart, correct?
> 2) Is it true that this needs to be specified for each node of a clustered
> environment?
>
> Thanks again!
>
> [1] https://nifi.apache.org/registry.html
>
>
>
> --
> Sent from: http://apache-nifi-developer-list.39713.n7.nabble.com/
Re: Custom properties file for environmental promotion
Posted by wildo <wi...@moserit.com>.
Great info Bryan- thanks!
Regarding my first question, I talked to our admins and we only have one NIC
anyway. So there is no need for me to limit it, and thus I don't have a need
to use EL to discover the NIC. So that's good.
Regarding the registry stuff, I found this [1] document which looks
FANTASTIC. But I'm not able to find when/if this stuff will be released. My
understanding is that it is not yet released, and therefore I'm assuming
that specifying a custom.properties file via the
nifi.variable.registry.properties is still the preferred method.
Additionally, this will mean that:
1) Changes to this file require a service restart, correct?
2) Is it true that this needs to be specified for each node of a clustered
environment?
Thanks again!
[1] https://nifi.apache.org/registry.html
--
Sent from: http://apache-nifi-developer-list.39713.n7.nabble.com/
Re: Custom properties file for environmental promotion
Posted by Bryan Bende <bb...@gmail.com>.
Hello,
Regarding Remote Process Groups, this is definitely something that
needs to be improved. There is a JIRA to make the URL editable [1].
A significant amount of work has been done on the flow registry [2],
and this will become the primary way to deploy flows across
environments.
The typical scenario would be to save your dev flow to the registry,
and when importing it to QA or prod, you would then edit the RPG
(based on the JIRA to make it editable) to be the URL for that
environment.
After that the URL would be set for that environment and would not be
changed when upgrading to newer versions.
Hope this helps.
Thanks,
Bryan
[1] https://issues.apache.org/jira/browse/NIFI-4526
[2] https://github.com/apache/nifi-registry
On Tue, Nov 7, 2017 at 11:58 PM, wildo <wi...@moserit.com> wrote:
> We have nearly wrapped up our testing with out Nifi scripts in dev, and are
> now looking to push to QA. I found an article about creating a custom
> properties file in order to specify each of your environmental specific
> variables, and then specifying that file in nifi.properties at
> nifi.variable.registry.properties.
>
> This will work fine omitting two cases I can think of.
>
> 1) We have a number of ListenTCP processors which require the "local network
> interface" to be specified. I have read that Expression Language can access
> system properties, but I haven't seen any example about how to use this. Can
> anyone share how EL might be used to grab the local network interface for
> each environment automatically?
>
> 2) We use Remote Process Groups with Site-to-Site for load balancing. In the
> RPG, you have to specify an absolute url to the nodes in the remote site.
> The RPG doesn't indicate that EL is acceptable in this field. Can anyone
> chime in on the possibility of using EL to grab a property for the RPG url?
>
> Thanks!
>
>
>
> --
> Sent from: http://apache-nifi-developer-list.39713.n7.nabble.com/