Loading config properties from environment

[Andrew Serff <an...@acesinc.net>]

I have what I’m hoping is a simple question that I just can’t find the answer to. We run our NiFi in development, integration and production environments and in each environment we have different login properties needed for things like Mongo processors, JMS processors, etc. The username/password and url to log into mongo for instance is different in our dev environment and our productions environment. We have many of these processors within our flow and when we move it from one environment to another, we have to open the flow file and update all those processors.  We can do this brute force in the flow.xml file, but it’s still a very manual process going from one environment to another.

Is there a way to externalize these kinds of properties from the flow.xml? I’ve tried putting them into the nifi.conf and access them with something like ${mongo.username}, but that doesn’t work.  I don’t think the Mongo Processor supports expression language, but I’ve also tried this in something like the ListenTCP processor that does and it still doesn’t work.

It seems like this has to be a very common problem, so I hope I’ve just missed it somewhere in the docs. I can’t find anything that would help besides a ControllerService, but the processors would need to support it and the ones out of the box don’t seem to. I also don’t see how you’d pull properties from the nifi.conf file into those Controller Services either.

If anyone could help us make out deployments more streamlined, that would be great!

Thanks!
Andrew

Re: Loading config properties from environment

[Andrew Serff <an...@acesinc.net>]

Thanks for the info Joe! So you’re saying that pre-1.0 there isn’t a good way to do this besides the manual process we are doing now? We’ll look at the 1.0 beta as soon as we can. I did try out 1.0 about a month ago and wasn’t too fond of the UI updates, but I’ll try it out again and see what’s changed. 

The deterministic template export is very intriguing if it truly let us commit the flow file as straight xml. Doing distributed development with nifi right now is near impossible with no merging capabilities.  I’m a little concerned that “export” is in that feature name though.  It make it sound like you have to export the flow to some new file to get it to be in this more friendly format.  Is that true?  The first thing I had tried in the past was to just unzip the flow file and run it like that in development, but nifi didn’t like that. I always thought that would have been the easy solution was to run uncompressed in dev and compress it in production. But again, we’ll take a look. 

Thanks!
Andrew




On 8/5/16, 6:11 PM, "Joe Witt" <jo...@gmail.com> wrote:

>Andrew,
>
>You are right that it is a common request and desire and the
>dev/stagin/prod lifecycle you mention makes sense and is also common.
>
>In the upcoming nifi 1.0 release a variable registry has been
>introduced which will allow you to do exactly what you show above for
>any property of any  component (processor or controller service) that
>supports expression language. Several more processors have been
>updated to be able to take advantage of this but many more need to be.
>
>We can and should also keep making progress here well beyond what is
>in the upcoming 1.0 release.  We should make the variable registry
>editable through the UI and it will need to support sensitive
>properties.  The work in 1.0 lays that foundation nicely.  We should
>also be able to support things like cluster wide scope vs node
>specific scope.  Cluster wide scope examples would be things like
>passwords for external services and node specific scope would be
>things like directory locations on the local file system.
>
>Also aiding the lifecycle you mention is that in the 1.0 line we now
>have support for deterministic template export.  This means you can
>save templates into git and use normal diffing tools and get far more
>predictable results which also makes it easier to see what has
>changed.  We should also go further there to include graphical diffs
>(of the flow graph itself not just some text representation).  This
>deterministic template support combined with the variable registry
>means they're more portable and more useful.  Again, here too we want
>to do much more.
>
>We just kicked off a vote to release the 1.0-Beta and once that is out
>it would be great if you could give that a shot and see if that helps
>you make progress.
>
>Thanks
>Joe
>
>On Fri, Aug 5, 2016 at 6:23 PM, Andrew Serff <an...@acesinc.net> wrote:
>> I have what I’m hoping is a simple question that I just can’t find the
>> answer to. We run our NiFi in development, integration and production
>> environments and in each environment we have different login properties
>> needed for things like Mongo processors, JMS processors, etc. The
>> username/password and url to log into mongo for instance is different in our
>> dev environment and our productions environment. We have many of these
>> processors within our flow and when we move it from one environment to
>> another, we have to open the flow file and update all those processors.  We
>> can do this brute force in the flow.xml file, but it’s still a very manual
>> process going from one environment to another.
>>
>> Is there a way to externalize these kinds of properties from the flow.xml?
>> I’ve tried putting them into the nifi.conf and access them with something
>> like ${mongo.username}, but that doesn’t work.  I don’t think the Mongo
>> Processor supports expression language, but I’ve also tried this in
>> something like the ListenTCP processor that does and it still doesn’t work.
>>
>> It seems like this has to be a very common problem, so I hope I’ve just
>> missed it somewhere in the docs. I can’t find anything that would help
>> besides a ControllerService, but the processors would need to support it and
>> the ones out of the box don’t seem to. I also don’t see how you’d pull
>> properties from the nifi.conf file into those Controller Services either.
>>
>> If anyone could help us make out deployments more streamlined, that would be
>> great!
>>
>> Thanks!
>> Andrew

Re: Loading config properties from environment

[Joe Witt <jo...@gmail.com>]

Andrew,

You are right that it is a common request and desire and the
dev/stagin/prod lifecycle you mention makes sense and is also common.

In the upcoming nifi 1.0 release a variable registry has been
introduced which will allow you to do exactly what you show above for
any property of any  component (processor or controller service) that
supports expression language. Several more processors have been
updated to be able to take advantage of this but many more need to be.

We can and should also keep making progress here well beyond what is
in the upcoming 1.0 release.  We should make the variable registry
editable through the UI and it will need to support sensitive
properties.  The work in 1.0 lays that foundation nicely.  We should
also be able to support things like cluster wide scope vs node
specific scope.  Cluster wide scope examples would be things like
passwords for external services and node specific scope would be
things like directory locations on the local file system.

Also aiding the lifecycle you mention is that in the 1.0 line we now
have support for deterministic template export.  This means you can
save templates into git and use normal diffing tools and get far more
predictable results which also makes it easier to see what has
changed.  We should also go further there to include graphical diffs
(of the flow graph itself not just some text representation).  This
deterministic template support combined with the variable registry
means they're more portable and more useful.  Again, here too we want
to do much more.

We just kicked off a vote to release the 1.0-Beta and once that is out
it would be great if you could give that a shot and see if that helps
you make progress.

Thanks
Joe

On Fri, Aug 5, 2016 at 6:23 PM, Andrew Serff <an...@acesinc.net> wrote:
> I have what I’m hoping is a simple question that I just can’t find the
> answer to. We run our NiFi in development, integration and production
> environments and in each environment we have different login properties
> needed for things like Mongo processors, JMS processors, etc. The
> username/password and url to log into mongo for instance is different in our
> dev environment and our productions environment. We have many of these
> processors within our flow and when we move it from one environment to
> another, we have to open the flow file and update all those processors.  We
> can do this brute force in the flow.xml file, but it’s still a very manual
> process going from one environment to another.
>
> Is there a way to externalize these kinds of properties from the flow.xml?
> I’ve tried putting them into the nifi.conf and access them with something
> like ${mongo.username}, but that doesn’t work.  I don’t think the Mongo
> Processor supports expression language, but I’ve also tried this in
> something like the ListenTCP processor that does and it still doesn’t work.
>
> It seems like this has to be a very common problem, so I hope I’ve just
> missed it somewhere in the docs. I can’t find anything that would help
> besides a ControllerService, but the processors would need to support it and
> the ones out of the box don’t seem to. I also don’t see how you’d pull
> properties from the nifi.conf file into those Controller Services either.
>
> If anyone could help us make out deployments more streamlined, that would be
> great!
>
> Thanks!
> Andrew