You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@nifi.apache.org by Jonathan Meran <jo...@sonos.com> on 2018/10/24 18:46:10 UTC
Re: [External] Re: Expression language and NiFi registry
Hello there,
Regarding expression language properties, why aren't all or most properties enabled by default.
Is there substantial processor overhead on these?
Thanks,
Jon
On 10/24/18, 1:49 PM, "Joe Witt" <jo...@gmail.com> wrote:
Mark
There are two scenarios here to discuss:
1) What do to with sensitive properties
2) What to do with properties that dont allow expression language statements
For #1 we dont send those properties into the registry and they are
set (properly) in each environment where the secret belongs. All good
as that doesn't change the flow definition.
For #2 we should just look at which properties are causing trouble and
see about expression language enabling them. So please share
precisely which ones you're hitting where that would help you out and
lets see what can be done.
Thanks
On Wed, Oct 24, 2018 at 1:39 PM Mark Littleton
<ma...@hotmail.co.uk> wrote:
>
> Hi Everyone,
>
> I'm currently doing a lot of work with Nifi and recently we have been trying to come up with a solution to a problem. We installed Nifi registry backed by our Git repository for versioning our flows. This has worked out great for us as we can now track the version of our flows correctly and make sure they are backed up in source control.
>
> However when we want to do deployment between our Development Nifi cluster and our Qa Nifi cluster we have to ofcourse change some values. These could be amqp queues, directories on the file system etc.
>
> So ofcourse we use variables so that we can configure the values without it being detected as a change to the flow. A problem arises however when we need to configure an option that does not support expression language. For example the host name of the amqp processors.
>
> This leaves us in a situation where a change to the flow is detected. The only real option I have as far as I can see is to clone the flows and have one for each environment which I don't like at all.
>
> Is anyone else struggling with similar issues. If so how are you handling It?
>
> Sent from my Sony Xperia™ smartphone
Re: [External] Re: Expression language and NiFi registry
Posted by Andy LoPresto <al...@gmail.com>.
Currently variables are not encrypted or protected sufficiently, so this behavior is discouraged and you should at least be aware of those risks. There are numerous discussions, usually on PRs, where we explain that there are difficulties in protecting those values and parsing EL in sensitive properties — for example, a password “abc${def}”. Is that a literal 9 character password, or should the value of variable “def” be appended to the literal “abc”?
Andy LoPresto
alopresto@apache.org
alopresto.apache@gmail.com
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
> On Oct 27, 2018, at 04:40, Jon Logan <jm...@buffalo.edu> wrote:
>
> Speaking of EL and Secrets -- we have been setting some secret values as EL
> expressions to substitute in properties for passwords and such. If we
> switch to NiFi registry, this would still be considered a sensitive value
> and not be stored? Is there a better way to handle secret distribution?
>
>> On Fri, Oct 26, 2018 at 3:38 PM Michael Moser <mo...@gmail.com> wrote:
>>
>> For the specific case of "host name of the AMQP processors", EL support for
>> that property was added to NiFi in NIFI-5489 [1] which will be released
>> with NiFi 1.8.0 (available real soon!)
>>
>> [1] - https://issues.apache.org/jira/browse/NIFI-5489
>>
>> -- Mike
>>
>>
>> On Wed, Oct 24, 2018 at 3:06 PM Pierre Villard <
>> pierre.villard.fr@gmail.com>
>> wrote:
>>
>>> Hi there,
>>>
>>> There is a JIRA [1] about this idea and, as Joe said, we need to
>> carefully
>>> think about the implications. Unless we can do something nice and
>> backward
>>> compatible, that could be something for NiFi 2.0.
>>>
>>> [1] https://issues.apache.org/jira/browse/NIFI-5367
>>>
>>> Pierre
>>>
>>>> Le mer. 24 oct. 2018 à 20:59, Joe Witt <jo...@gmail.com> a écrit :
>>>>
>>>> Jon,
>>>>
>>>> There is very little overhead really. The answer more generally has
>>>> to do with history. Expression language was added later in the game
>>>> and there can be different scopes to what a given EL statement has
>>>> access to. Some EL are evaluated against flowfiles and some arent.
>>>> We didn't have a good way to show users this information though this
>>>> has recently improved. It also has to do with 'evaluation time'
>>>> meaning processors might evaluate some properties frequently/while
>>>> executing a task (as would be the case when EL statements have a given
>>>> flowfile in scope) or during enabling/scheduling such as pulling a
>>>> property value they intend to honor throughout an entire runtime.
>>>>
>>>> I suspect at this point that we can make most properties (not
>>>> enumerations most likely) EL enabled. But, each one requires
>>>> review/consideration to ensure the processor is coded to use it
>>>> correctly.
>>>>
>>>> Alternatively, we could explore making all text entry fields allow
>>>> users to explicitly state 'this is a variable and here is the variable
>>>> name' (but in a cool UI way) and then the actual value used would come
>>>> from the variable registry for that process group. This would be
>>>> something we can do across the board as it wouldnt' change the
>>>> processor's logic/behavior at all. Variables get set at and retained
>>>> throughout the life of a processor run. If a used variable is changed
>>>> we'd automatically stop/start each impacted component/processor so it
>>>> wouldnt' require any additional review/changes in processor
>>>> properties.
>>>>
>>>> ThanksOn Wed, Oct 24, 2018 at 2:46 PM Jonathan Meran
>>>> <jo...@sonos.com> wrote:
>>>>>
>>>>> Hello there,
>>>>> Regarding expression language properties, why aren't all or most
>>>> properties enabled by default.
>>>>>
>>>>> Is there substantial processor overhead on these?
>>>>>
>>>>> Thanks,
>>>>> Jon
>>>>>
>>>>> On 10/24/18, 1:49 PM, "Joe Witt" <jo...@gmail.com> wrote:
>>>>>
>>>>> Mark
>>>>>
>>>>> There are two scenarios here to discuss:
>>>>> 1) What do to with sensitive properties
>>>>> 2) What to do with properties that dont allow expression language
>>>> statements
>>>>>
>>>>> For #1 we dont send those properties into the registry and they
>> are
>>>>> set (properly) in each environment where the secret belongs. All
>>>> good
>>>>> as that doesn't change the flow definition.
>>>>>
>>>>> For #2 we should just look at which properties are causing
>> trouble
>>>> and
>>>>> see about expression language enabling them. So please share
>>>>> precisely which ones you're hitting where that would help you out
>>> and
>>>>> lets see what can be done.
>>>>>
>>>>> Thanks
>>>>> On Wed, Oct 24, 2018 at 1:39 PM Mark Littleton
>>>>> <ma...@hotmail.co.uk> wrote:
>>>>>>
>>>>>> Hi Everyone,
>>>>>>
>>>>>> I'm currently doing a lot of work with Nifi and recently we
>> have
>>>> been trying to come up with a solution to a problem. We installed Nifi
>>>> registry backed by our Git repository for versioning our flows. This
>> has
>>>> worked out great for us as we can now track the version of our flows
>>>> correctly and make sure they are backed up in source control.
>>>>>>
>>>>>> However when we want to do deployment between our Development
>>> Nifi
>>>> cluster and our Qa Nifi cluster we have to ofcourse change some values.
>>>> These could be amqp queues, directories on the file system etc.
>>>>>>
>>>>>> So ofcourse we use variables so that we can configure the
>> values
>>>> without it being detected as a change to the flow. A problem arises
>>> however
>>>> when we need to configure an option that does not support expression
>>>> language. For example the host name of the amqp processors.
>>>>>>
>>>>>> This leaves us in a situation where a change to the flow is
>>>> detected. The only real option I have as far as I can see is to clone
>> the
>>>> flows and have one for each environment which I don't like at all.
>>>>>>
>>>>>> Is anyone else struggling with similar issues. If so how are
>> you
>>>> handling It?
>>>>>>
>>>>>> Sent from my Sony Xperia™ smartphone
>>>>>
>>>>>
>>>>
>>>
>>
Re: [External] Re: Expression language and NiFi registry
Posted by Jon Logan <jm...@buffalo.edu>.
Speaking of EL and Secrets -- we have been setting some secret values as EL
expressions to substitute in properties for passwords and such. If we
switch to NiFi registry, this would still be considered a sensitive value
and not be stored? Is there a better way to handle secret distribution?
On Fri, Oct 26, 2018 at 3:38 PM Michael Moser <mo...@gmail.com> wrote:
> For the specific case of "host name of the AMQP processors", EL support for
> that property was added to NiFi in NIFI-5489 [1] which will be released
> with NiFi 1.8.0 (available real soon!)
>
> [1] - https://issues.apache.org/jira/browse/NIFI-5489
>
> -- Mike
>
>
> On Wed, Oct 24, 2018 at 3:06 PM Pierre Villard <
> pierre.villard.fr@gmail.com>
> wrote:
>
> > Hi there,
> >
> > There is a JIRA [1] about this idea and, as Joe said, we need to
> carefully
> > think about the implications. Unless we can do something nice and
> backward
> > compatible, that could be something for NiFi 2.0.
> >
> > [1] https://issues.apache.org/jira/browse/NIFI-5367
> >
> > Pierre
> >
> > Le mer. 24 oct. 2018 à 20:59, Joe Witt <jo...@gmail.com> a écrit :
> >
> > > Jon,
> > >
> > > There is very little overhead really. The answer more generally has
> > > to do with history. Expression language was added later in the game
> > > and there can be different scopes to what a given EL statement has
> > > access to. Some EL are evaluated against flowfiles and some arent.
> > > We didn't have a good way to show users this information though this
> > > has recently improved. It also has to do with 'evaluation time'
> > > meaning processors might evaluate some properties frequently/while
> > > executing a task (as would be the case when EL statements have a given
> > > flowfile in scope) or during enabling/scheduling such as pulling a
> > > property value they intend to honor throughout an entire runtime.
> > >
> > > I suspect at this point that we can make most properties (not
> > > enumerations most likely) EL enabled. But, each one requires
> > > review/consideration to ensure the processor is coded to use it
> > > correctly.
> > >
> > > Alternatively, we could explore making all text entry fields allow
> > > users to explicitly state 'this is a variable and here is the variable
> > > name' (but in a cool UI way) and then the actual value used would come
> > > from the variable registry for that process group. This would be
> > > something we can do across the board as it wouldnt' change the
> > > processor's logic/behavior at all. Variables get set at and retained
> > > throughout the life of a processor run. If a used variable is changed
> > > we'd automatically stop/start each impacted component/processor so it
> > > wouldnt' require any additional review/changes in processor
> > > properties.
> > >
> > > ThanksOn Wed, Oct 24, 2018 at 2:46 PM Jonathan Meran
> > > <jo...@sonos.com> wrote:
> > > >
> > > > Hello there,
> > > > Regarding expression language properties, why aren't all or most
> > > properties enabled by default.
> > > >
> > > > Is there substantial processor overhead on these?
> > > >
> > > > Thanks,
> > > > Jon
> > > >
> > > > On 10/24/18, 1:49 PM, "Joe Witt" <jo...@gmail.com> wrote:
> > > >
> > > > Mark
> > > >
> > > > There are two scenarios here to discuss:
> > > > 1) What do to with sensitive properties
> > > > 2) What to do with properties that dont allow expression language
> > > statements
> > > >
> > > > For #1 we dont send those properties into the registry and they
> are
> > > > set (properly) in each environment where the secret belongs. All
> > > good
> > > > as that doesn't change the flow definition.
> > > >
> > > > For #2 we should just look at which properties are causing
> trouble
> > > and
> > > > see about expression language enabling them. So please share
> > > > precisely which ones you're hitting where that would help you out
> > and
> > > > lets see what can be done.
> > > >
> > > > Thanks
> > > > On Wed, Oct 24, 2018 at 1:39 PM Mark Littleton
> > > > <ma...@hotmail.co.uk> wrote:
> > > > >
> > > > > Hi Everyone,
> > > > >
> > > > > I'm currently doing a lot of work with Nifi and recently we
> have
> > > been trying to come up with a solution to a problem. We installed Nifi
> > > registry backed by our Git repository for versioning our flows. This
> has
> > > worked out great for us as we can now track the version of our flows
> > > correctly and make sure they are backed up in source control.
> > > > >
> > > > > However when we want to do deployment between our Development
> > Nifi
> > > cluster and our Qa Nifi cluster we have to ofcourse change some values.
> > > These could be amqp queues, directories on the file system etc.
> > > > >
> > > > > So ofcourse we use variables so that we can configure the
> values
> > > without it being detected as a change to the flow. A problem arises
> > however
> > > when we need to configure an option that does not support expression
> > > language. For example the host name of the amqp processors.
> > > > >
> > > > > This leaves us in a situation where a change to the flow is
> > > detected. The only real option I have as far as I can see is to clone
> the
> > > flows and have one for each environment which I don't like at all.
> > > > >
> > > > > Is anyone else struggling with similar issues. If so how are
> you
> > > handling It?
> > > > >
> > > > > Sent from my Sony Xperia™ smartphone
> > > >
> > > >
> > >
> >
>
Re: [External] Re: Expression language and NiFi registry
Posted by Michael Moser <mo...@gmail.com>.
For the specific case of "host name of the AMQP processors", EL support for
that property was added to NiFi in NIFI-5489 [1] which will be released
with NiFi 1.8.0 (available real soon!)
[1] - https://issues.apache.org/jira/browse/NIFI-5489
-- Mike
On Wed, Oct 24, 2018 at 3:06 PM Pierre Villard <pi...@gmail.com>
wrote:
> Hi there,
>
> There is a JIRA [1] about this idea and, as Joe said, we need to carefully
> think about the implications. Unless we can do something nice and backward
> compatible, that could be something for NiFi 2.0.
>
> [1] https://issues.apache.org/jira/browse/NIFI-5367
>
> Pierre
>
> Le mer. 24 oct. 2018 à 20:59, Joe Witt <jo...@gmail.com> a écrit :
>
> > Jon,
> >
> > There is very little overhead really. The answer more generally has
> > to do with history. Expression language was added later in the game
> > and there can be different scopes to what a given EL statement has
> > access to. Some EL are evaluated against flowfiles and some arent.
> > We didn't have a good way to show users this information though this
> > has recently improved. It also has to do with 'evaluation time'
> > meaning processors might evaluate some properties frequently/while
> > executing a task (as would be the case when EL statements have a given
> > flowfile in scope) or during enabling/scheduling such as pulling a
> > property value they intend to honor throughout an entire runtime.
> >
> > I suspect at this point that we can make most properties (not
> > enumerations most likely) EL enabled. But, each one requires
> > review/consideration to ensure the processor is coded to use it
> > correctly.
> >
> > Alternatively, we could explore making all text entry fields allow
> > users to explicitly state 'this is a variable and here is the variable
> > name' (but in a cool UI way) and then the actual value used would come
> > from the variable registry for that process group. This would be
> > something we can do across the board as it wouldnt' change the
> > processor's logic/behavior at all. Variables get set at and retained
> > throughout the life of a processor run. If a used variable is changed
> > we'd automatically stop/start each impacted component/processor so it
> > wouldnt' require any additional review/changes in processor
> > properties.
> >
> > ThanksOn Wed, Oct 24, 2018 at 2:46 PM Jonathan Meran
> > <jo...@sonos.com> wrote:
> > >
> > > Hello there,
> > > Regarding expression language properties, why aren't all or most
> > properties enabled by default.
> > >
> > > Is there substantial processor overhead on these?
> > >
> > > Thanks,
> > > Jon
> > >
> > > On 10/24/18, 1:49 PM, "Joe Witt" <jo...@gmail.com> wrote:
> > >
> > > Mark
> > >
> > > There are two scenarios here to discuss:
> > > 1) What do to with sensitive properties
> > > 2) What to do with properties that dont allow expression language
> > statements
> > >
> > > For #1 we dont send those properties into the registry and they are
> > > set (properly) in each environment where the secret belongs. All
> > good
> > > as that doesn't change the flow definition.
> > >
> > > For #2 we should just look at which properties are causing trouble
> > and
> > > see about expression language enabling them. So please share
> > > precisely which ones you're hitting where that would help you out
> and
> > > lets see what can be done.
> > >
> > > Thanks
> > > On Wed, Oct 24, 2018 at 1:39 PM Mark Littleton
> > > <ma...@hotmail.co.uk> wrote:
> > > >
> > > > Hi Everyone,
> > > >
> > > > I'm currently doing a lot of work with Nifi and recently we have
> > been trying to come up with a solution to a problem. We installed Nifi
> > registry backed by our Git repository for versioning our flows. This has
> > worked out great for us as we can now track the version of our flows
> > correctly and make sure they are backed up in source control.
> > > >
> > > > However when we want to do deployment between our Development
> Nifi
> > cluster and our Qa Nifi cluster we have to ofcourse change some values.
> > These could be amqp queues, directories on the file system etc.
> > > >
> > > > So ofcourse we use variables so that we can configure the values
> > without it being detected as a change to the flow. A problem arises
> however
> > when we need to configure an option that does not support expression
> > language. For example the host name of the amqp processors.
> > > >
> > > > This leaves us in a situation where a change to the flow is
> > detected. The only real option I have as far as I can see is to clone the
> > flows and have one for each environment which I don't like at all.
> > > >
> > > > Is anyone else struggling with similar issues. If so how are you
> > handling It?
> > > >
> > > > Sent from my Sony Xperia™ smartphone
> > >
> > >
> >
>
Re: [External] Re: Expression language and NiFi registry
Posted by Pierre Villard <pi...@gmail.com>.
Hi there,
There is a JIRA [1] about this idea and, as Joe said, we need to carefully
think about the implications. Unless we can do something nice and backward
compatible, that could be something for NiFi 2.0.
[1] https://issues.apache.org/jira/browse/NIFI-5367
Pierre
Le mer. 24 oct. 2018 à 20:59, Joe Witt <jo...@gmail.com> a écrit :
> Jon,
>
> There is very little overhead really. The answer more generally has
> to do with history. Expression language was added later in the game
> and there can be different scopes to what a given EL statement has
> access to. Some EL are evaluated against flowfiles and some arent.
> We didn't have a good way to show users this information though this
> has recently improved. It also has to do with 'evaluation time'
> meaning processors might evaluate some properties frequently/while
> executing a task (as would be the case when EL statements have a given
> flowfile in scope) or during enabling/scheduling such as pulling a
> property value they intend to honor throughout an entire runtime.
>
> I suspect at this point that we can make most properties (not
> enumerations most likely) EL enabled. But, each one requires
> review/consideration to ensure the processor is coded to use it
> correctly.
>
> Alternatively, we could explore making all text entry fields allow
> users to explicitly state 'this is a variable and here is the variable
> name' (but in a cool UI way) and then the actual value used would come
> from the variable registry for that process group. This would be
> something we can do across the board as it wouldnt' change the
> processor's logic/behavior at all. Variables get set at and retained
> throughout the life of a processor run. If a used variable is changed
> we'd automatically stop/start each impacted component/processor so it
> wouldnt' require any additional review/changes in processor
> properties.
>
> ThanksOn Wed, Oct 24, 2018 at 2:46 PM Jonathan Meran
> <jo...@sonos.com> wrote:
> >
> > Hello there,
> > Regarding expression language properties, why aren't all or most
> properties enabled by default.
> >
> > Is there substantial processor overhead on these?
> >
> > Thanks,
> > Jon
> >
> > On 10/24/18, 1:49 PM, "Joe Witt" <jo...@gmail.com> wrote:
> >
> > Mark
> >
> > There are two scenarios here to discuss:
> > 1) What do to with sensitive properties
> > 2) What to do with properties that dont allow expression language
> statements
> >
> > For #1 we dont send those properties into the registry and they are
> > set (properly) in each environment where the secret belongs. All
> good
> > as that doesn't change the flow definition.
> >
> > For #2 we should just look at which properties are causing trouble
> and
> > see about expression language enabling them. So please share
> > precisely which ones you're hitting where that would help you out and
> > lets see what can be done.
> >
> > Thanks
> > On Wed, Oct 24, 2018 at 1:39 PM Mark Littleton
> > <ma...@hotmail.co.uk> wrote:
> > >
> > > Hi Everyone,
> > >
> > > I'm currently doing a lot of work with Nifi and recently we have
> been trying to come up with a solution to a problem. We installed Nifi
> registry backed by our Git repository for versioning our flows. This has
> worked out great for us as we can now track the version of our flows
> correctly and make sure they are backed up in source control.
> > >
> > > However when we want to do deployment between our Development Nifi
> cluster and our Qa Nifi cluster we have to ofcourse change some values.
> These could be amqp queues, directories on the file system etc.
> > >
> > > So ofcourse we use variables so that we can configure the values
> without it being detected as a change to the flow. A problem arises however
> when we need to configure an option that does not support expression
> language. For example the host name of the amqp processors.
> > >
> > > This leaves us in a situation where a change to the flow is
> detected. The only real option I have as far as I can see is to clone the
> flows and have one for each environment which I don't like at all.
> > >
> > > Is anyone else struggling with similar issues. If so how are you
> handling It?
> > >
> > > Sent from my Sony Xperia™ smartphone
> >
> >
>
Re: [External] Re: Expression language and NiFi registry
Posted by Joe Witt <jo...@gmail.com>.
Jon,
There is very little overhead really. The answer more generally has
to do with history. Expression language was added later in the game
and there can be different scopes to what a given EL statement has
access to. Some EL are evaluated against flowfiles and some arent.
We didn't have a good way to show users this information though this
has recently improved. It also has to do with 'evaluation time'
meaning processors might evaluate some properties frequently/while
executing a task (as would be the case when EL statements have a given
flowfile in scope) or during enabling/scheduling such as pulling a
property value they intend to honor throughout an entire runtime.
I suspect at this point that we can make most properties (not
enumerations most likely) EL enabled. But, each one requires
review/consideration to ensure the processor is coded to use it
correctly.
Alternatively, we could explore making all text entry fields allow
users to explicitly state 'this is a variable and here is the variable
name' (but in a cool UI way) and then the actual value used would come
from the variable registry for that process group. This would be
something we can do across the board as it wouldnt' change the
processor's logic/behavior at all. Variables get set at and retained
throughout the life of a processor run. If a used variable is changed
we'd automatically stop/start each impacted component/processor so it
wouldnt' require any additional review/changes in processor
properties.
ThanksOn Wed, Oct 24, 2018 at 2:46 PM Jonathan Meran
<jo...@sonos.com> wrote:
>
> Hello there,
> Regarding expression language properties, why aren't all or most properties enabled by default.
>
> Is there substantial processor overhead on these?
>
> Thanks,
> Jon
>
> On 10/24/18, 1:49 PM, "Joe Witt" <jo...@gmail.com> wrote:
>
> Mark
>
> There are two scenarios here to discuss:
> 1) What do to with sensitive properties
> 2) What to do with properties that dont allow expression language statements
>
> For #1 we dont send those properties into the registry and they are
> set (properly) in each environment where the secret belongs. All good
> as that doesn't change the flow definition.
>
> For #2 we should just look at which properties are causing trouble and
> see about expression language enabling them. So please share
> precisely which ones you're hitting where that would help you out and
> lets see what can be done.
>
> Thanks
> On Wed, Oct 24, 2018 at 1:39 PM Mark Littleton
> <ma...@hotmail.co.uk> wrote:
> >
> > Hi Everyone,
> >
> > I'm currently doing a lot of work with Nifi and recently we have been trying to come up with a solution to a problem. We installed Nifi registry backed by our Git repository for versioning our flows. This has worked out great for us as we can now track the version of our flows correctly and make sure they are backed up in source control.
> >
> > However when we want to do deployment between our Development Nifi cluster and our Qa Nifi cluster we have to ofcourse change some values. These could be amqp queues, directories on the file system etc.
> >
> > So ofcourse we use variables so that we can configure the values without it being detected as a change to the flow. A problem arises however when we need to configure an option that does not support expression language. For example the host name of the amqp processors.
> >
> > This leaves us in a situation where a change to the flow is detected. The only real option I have as far as I can see is to clone the flows and have one for each environment which I don't like at all.
> >
> > Is anyone else struggling with similar issues. If so how are you handling It?
> >
> > Sent from my Sony Xperia™ smartphone
>
>