You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@nifi.apache.org by ddewaele <dd...@gmail.com> on 2016/12/12 08:04:20 UTC
Making FlowFiles environment independent
We have a flowfile that contains a number of environment specific values
(ports / hostnames / .....).
Am I correct in saying that there is no immediate variable registry
somewhere in nifi, and that all of these environment specific items need to
be passed as environment variables or java system properties ?
I understand that the nifi expression language allows us to retrieve
environment variables / system properties, but a number of processor don't
support the expression language for fields that do contain environment
specific values (like the PutTCP processor hostname property).
How should we go about updating those ?
--
View this message in context: http://apache-nifi-users-list.2361937.n4.nabble.com/Making-FlowFiles-environment-independent-tp409.html
Sent from the Apache NiFi Users List mailing list archive at Nabble.com.
Re: Making FlowFiles environment independent
Posted by ddewaele <dd...@gmail.com>.
Russell Bateman-2 wrote
> Without completely understanding your scenario, may I suggest you insert
> UpdateAttributesto do this before PutTCP? It supports Expression Language.
I think I used the term "flowfile" out of context here... I meant a flow
template that needs to be deployed on different environments where certain
processor attributes (like a tcp hostname) need to be externalized, as they
are different for each environment (dev/prod/...)
--
View this message in context: http://apache-nifi-users-list.2361937.n4.nabble.com/Making-FlowFiles-environment-independent-tp409p416.html
Sent from the Apache NiFi Users List mailing list archive at Nabble.com.
Re: Making FlowFiles environment independent
Posted by Bryan Bende <bb...@gmail.com>.
There is no reason it can't support EL without a flow file, I created this
JIRA: https://issues.apache.org/jira/browse/NIFI-3231
EL per flow file, meaning the flow file specifies the host and port, would
require a fairly significant change to how the processor works internally,
but it doesn't sound like that is what we are talking about here.
On Tue, Dec 20, 2016 at 7:05 AM, ddewaele <dd...@gmail.com> wrote:
> Coming back to PutTCP, is there a reason why the hostname property doesn't
> support EL ?
>
> In both cases you mentioned you would like the option to externalise it or
> make it dynamic.
>
> Are there other ways of injecting a hostname in the PutTCP processor ?
>
>
>
> --
> View this message in context: http://apache-nifi-users-list.
> 2361937.n4.nabble.com/Making-FlowFiles-environment-
> independent-tp409p498.html
> Sent from the Apache NiFi Users List mailing list archive at Nabble.com.
>
Re: Making FlowFiles environment independent
Posted by ddewaele <dd...@gmail.com>.
Coming back to PutTCP, is there a reason why the hostname property doesn't
support EL ?
In both cases you mentioned you would like the option to externalise it or
make it dynamic.
Are there other ways of injecting a hostname in the PutTCP processor ?
--
View this message in context: http://apache-nifi-users-list.2361937.n4.nabble.com/Making-FlowFiles-environment-independent-tp409p498.html
Sent from the Apache NiFi Users List mailing list archive at Nabble.com.
Re: Making FlowFiles environment independent
Posted by ddewaele <dd...@gmail.com>.
Thanks a lot ... I'll take a look at the variable registry.
In my case, I want to have a single flow definition (flow template) that we
can deploy in dev / prod, and have an easy way to externalise environment
specific stuff in the flow (like a tcp hostname / port)
I tried passing a system property to the PutTCP hostname property but that
didn't work (When I used ${mySystemPropertyName} the processor didn't
resolve the mySystemPropertyName but simply used "${mySystemPropertyName}"
as a string). I assume this is because the PutTCP hostname property doesn't
support expression language ?
--
View this message in context: http://apache-nifi-users-list.2361937.n4.nabble.com/Making-FlowFiles-environment-independent-tp409p415.html
Sent from the Apache NiFi Users List mailing list archive at Nabble.com.
Re: Making FlowFiles environment independent
Posted by ddewaele <dd...@gmail.com>.
Works like a charm ! Thanks.
Bryan Bende wrote
> There is a concept of a variable registry which is described in the Admin
> Guide here [1], but it is still based on expression language.
>
> For your use case with PutTCP, are you looking to have the hostname
> paramterized across environments (dev vs prod) or are you looking to
> connect to a different host per flow file?
>
> [1]
> https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#custom_properties
--
View this message in context: http://apache-nifi-users-list.2361937.n4.nabble.com/Making-FlowFiles-environment-independent-tp409p487.html
Sent from the Apache NiFi Users List mailing list archive at Nabble.com.
Re: Making FlowFiles environment independent
Posted by Bryan Bende <bb...@gmail.com>.
There is a concept of a variable registry which is described in the Admin
Guide here [1], but it is still based on expression language.
For your use case with PutTCP, are you looking to have the hostname
paramterized across environments (dev vs prod) or are you looking to
connect to a different host per flow file?
[1]
https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#custom_properties
On Mon, Dec 12, 2016 at 10:05 AM, Russell Bateman <ru...@windofkeltia.com>
wrote:
> Without completely understanding your scenario, may I suggest you insert
> UpdateAttributes to do this before PutTCP? It supports Expression
> Language.
>
> Hope this helps.
>
> Russ
>
> On 12/12/2016 01:04 AM, ddewaele wrote:
>
> We have a flowfile that contains a number of environment specific values
> (ports / hostnames / .....).
>
> Am I correct in saying that there is no immediate variable registry
> somewhere in nifi, and that all of these environment specific items need to
> be passed as environment variables or java system properties ?
>
> I understand that the nifi expression language allows us to retrieve
> environment variables / system properties, but a number of processor don't
> support the expression language for fields that do contain environment
> specific values (like the PutTCP processor hostname property).
>
> How should we go about updating those ?
>
>
>
> --
> View this message in context: http://apache-nifi-users-list.2361937.n4.nabble.com/Making-FlowFiles-environment-independent-tp409.html
> Sent from the Apache NiFi Users List mailing list archive at Nabble.com.
>
>
>
Re: Making FlowFiles environment independent
Posted by Russell Bateman <ru...@windofkeltia.com>.
Without completely understanding your scenario, may I suggest you insert
UpdateAttributesto do this before PutTCP? It supports Expression Language.
Hope this helps.
Russ
On 12/12/2016 01:04 AM, ddewaele wrote:
> We have a flowfile that contains a number of environment specific values
> (ports / hostnames / .....).
>
> Am I correct in saying that there is no immediate variable registry
> somewhere in nifi, and that all of these environment specific items need to
> be passed as environment variables or java system properties ?
>
> I understand that the nifi expression language allows us to retrieve
> environment variables / system properties, but a number of processor don't
> support the expression language for fields that do contain environment
> specific values (like the PutTCP processor hostname property).
>
> How should we go about updating those ?
>
>
>
> --
> View this message in context: http://apache-nifi-users-list.2361937.n4.nabble.com/Making-FlowFiles-environment-independent-tp409.html
> Sent from the Apache NiFi Users List mailing list archive at Nabble.com.