You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@nifi.apache.org by Chris Lundeberg <cl...@1904labs.com> on 2019/11/13 16:17:13 UTC
Parameters within password/passphrase fields
Hi all,
Recently we upgraded to Nifi 1.10 to take advantage of parameters and using
those within our deployment process (using the registry). I am seeing some
strange behavior and want to confirm my understanding of how this new
feature works, before digging deeper.
*Some general background info:*
I have a three node cluster running Nifi 1.10. Its security enabled, but I
am using an admin user with full rights to view/modify policies. Its
running on linux and using java 8.
*Producing the issue:*
As a test, I moved over some existing flows from our 1.9.2 cluster (same
specs and user rights as we build everything with puppet). A few of the
processors are pulling and pushing to SFTP, for which we use an RSA key, a
known hosts file and the "paraphrase" property descriptor is populated with
a password. I converted that password over to a parameter within the
master processor group, set the "Process Group Parameter Context" for the
processor group this processor resides and saved everything. After that
was done, I was able to access the parameter within #{ + ctrl + space. I
choose the correct one, closed in the curly braces and saved the
processor. I noticed immediately after saving and re-opening that the new
options icon to the right of the property descriptor reverted back from
"access the parameters" to "convert to a parameter". I didn't think much
of it, besides maybe a small bug in the UI (Although that does save
correctly when you set a non sensitive field). When I started the
processor, it immediately failed with an authentication error. I tried
setting a few different parameters, manually typing in #{ftp.password} into
the paraphrase and even setting a new parameter directly from the helper
icon within the processor. That didn't seem to change the behavior.
Lastly I went back into the processor and removed the "Sensitive value
set" and typed the password in plain text and saved. It worked fine after
that.
I thought it may have been something off with our policies or even install,
so I installed Nifi locally and experienced the same issue. It seems as
though the actual value is not being evaluated correctly when passed
into/through the processor for sensitive values.
Has anyone experienced this before?
Thanks!
Chris Lundeberg
Re: Parameters within password/passphrase fields
Posted by Bahlul Haider <ba...@gmail.com>.
I was facing the same issue in PutAzureBlobStorage 1.10.0. Sensitive
secrets from parameter context are being evaluated to null/empty string.
Thank you very much for fixing the issue in this Pull Request
<https://github.com/apache/nifi/pull/3889>.
I will build the source code for resolving the error in my nifi cluster
until a new version is released. Out of curiosity, when do we expect next
nifi minor version to be released?
Thanks,
Bahlul
On Wed, Nov 13, 2019 at 2:26 PM Robert Fellows <ro...@gmail.com>
wrote:
> Chris,
> I was able to reproduce your issue so I just logged the issue. You can
> track it here: https://issues.apache.org/jira/browse/NIFI-6868. Feel free
> to add any more context/comments as you like.
>
> - Rob
>
> On Wed, Nov 13, 2019 at 3:55 PM Robert Fellows <ro...@gmail.com>
> wrote:
>
> > Chris,
> > I think there are 2 things happening here. First, I think there might
> be
> > a bug in how sensitive parameters are evaluated on the backend and it is
> > causing your issue. Please log an issue for this.
> >
> > Second, the UI is a bit confusing in terms of the availability of the
> > "Convert to parameter" and "Go to Parameter" actions for sensitive
> > parameter references in properties. However, this is due to the nature of
> > sensitive parameters/properties. The actual value of the sensitive
> property
> > is only ever known to the UI when the user enters the value. Once
> applied,
> > those values are never sent back to the UI. The UI knows there is a value
> > set, but it has no idea if it contains a parameter reference or not. The
> > "Go To Parameter" action is only available when the UI can parse the
> value
> > of the property and it matches the '#{...}' syntax format. Since the
> values
> > is not sent to the UI, it can't parse the value to know if it is a
> > parameter reference or not. The "Convert to parameter" action option is
> > available just as a shortcut to create a new parameter, it is hidden when
> > it is known that the value already references a parameter. This may be
> > improved in the future, but it was implemented this way intentionally.
> >
> > - Rob
> >
> > On Wed, Nov 13, 2019 at 11:17 AM Chris Lundeberg <
> clundeberg@1904labs.com>
> > wrote:
> >
> >> Hi all,
> >>
> >> Recently we upgraded to Nifi 1.10 to take advantage of parameters and
> >> using
> >> those within our deployment process (using the registry). I am seeing
> >> some
> >> strange behavior and want to confirm my understanding of how this new
> >> feature works, before digging deeper.
> >>
> >> *Some general background info:*
> >> I have a three node cluster running Nifi 1.10. Its security enabled,
> but
> >> I
> >> am using an admin user with full rights to view/modify policies. Its
> >> running on linux and using java 8.
> >>
> >> *Producing the issue:*
> >> As a test, I moved over some existing flows from our 1.9.2 cluster (same
> >> specs and user rights as we build everything with puppet). A few of the
> >> processors are pulling and pushing to SFTP, for which we use an RSA
> key, a
> >> known hosts file and the "paraphrase" property descriptor is populated
> >> with
> >> a password. I converted that password over to a parameter within the
> >> master processor group, set the "Process Group Parameter Context" for
> the
> >> processor group this processor resides and saved everything. After that
> >> was done, I was able to access the parameter within #{ + ctrl + space.
> I
> >> choose the correct one, closed in the curly braces and saved the
> >> processor. I noticed immediately after saving and re-opening that the
> new
> >> options icon to the right of the property descriptor reverted back from
> >> "access the parameters" to "convert to a parameter". I didn't think
> much
> >> of it, besides maybe a small bug in the UI (Although that does save
> >> correctly when you set a non sensitive field). When I started the
> >> processor, it immediately failed with an authentication error. I tried
> >> setting a few different parameters, manually typing in #{ftp.password}
> >> into
> >> the paraphrase and even setting a new parameter directly from the helper
> >> icon within the processor. That didn't seem to change the behavior.
> >> Lastly I went back into the processor and removed the "Sensitive value
> >> set" and typed the password in plain text and saved. It worked fine
> after
> >> that.
> >>
> >> I thought it may have been something off with our policies or even
> >> install,
> >> so I installed Nifi locally and experienced the same issue. It seems as
> >> though the actual value is not being evaluated correctly when passed
> >> into/through the processor for sensitive values.
> >>
> >> Has anyone experienced this before?
> >>
> >> Thanks!
> >>
> >> Chris Lundeberg
> >>
> >
> >
> > --
> > -------------------------------
> > Rob Fellows
> >
>
>
> --
> -------------------------------
> Rob Fellows
>
--
Thanks,
Bahlul
Re: Parameters within password/passphrase fields
Posted by Chris Lundeberg <cl...@1904labs.com>.
Robert,
Thanks very much for the information about the UI and logging the ticket -
you beat me to the punch! I am glad you were able to reproduce.
Thanks again,
Chris Lundeberg
On Wed, Nov 13, 2019 at 4:26 PM Robert Fellows <ro...@gmail.com>
wrote:
> Chris,
> I was able to reproduce your issue so I just logged the issue. You can
> track it here: https://issues.apache.org/jira/browse/NIFI-6868. Feel free
> to add any more context/comments as you like.
>
> - Rob
>
> On Wed, Nov 13, 2019 at 3:55 PM Robert Fellows <ro...@gmail.com>
> wrote:
>
> > Chris,
> > I think there are 2 things happening here. First, I think there might
> be
> > a bug in how sensitive parameters are evaluated on the backend and it is
> > causing your issue. Please log an issue for this.
> >
> > Second, the UI is a bit confusing in terms of the availability of the
> > "Convert to parameter" and "Go to Parameter" actions for sensitive
> > parameter references in properties. However, this is due to the nature of
> > sensitive parameters/properties. The actual value of the sensitive
> property
> > is only ever known to the UI when the user enters the value. Once
> applied,
> > those values are never sent back to the UI. The UI knows there is a value
> > set, but it has no idea if it contains a parameter reference or not. The
> > "Go To Parameter" action is only available when the UI can parse the
> value
> > of the property and it matches the '#{...}' syntax format. Since the
> values
> > is not sent to the UI, it can't parse the value to know if it is a
> > parameter reference or not. The "Convert to parameter" action option is
> > available just as a shortcut to create a new parameter, it is hidden when
> > it is known that the value already references a parameter. This may be
> > improved in the future, but it was implemented this way intentionally.
> >
> > - Rob
> >
> > On Wed, Nov 13, 2019 at 11:17 AM Chris Lundeberg <
> clundeberg@1904labs.com>
> > wrote:
> >
> >> Hi all,
> >>
> >> Recently we upgraded to Nifi 1.10 to take advantage of parameters and
> >> using
> >> those within our deployment process (using the registry). I am seeing
> >> some
> >> strange behavior and want to confirm my understanding of how this new
> >> feature works, before digging deeper.
> >>
> >> *Some general background info:*
> >> I have a three node cluster running Nifi 1.10. Its security enabled,
> but
> >> I
> >> am using an admin user with full rights to view/modify policies. Its
> >> running on linux and using java 8.
> >>
> >> *Producing the issue:*
> >> As a test, I moved over some existing flows from our 1.9.2 cluster (same
> >> specs and user rights as we build everything with puppet). A few of the
> >> processors are pulling and pushing to SFTP, for which we use an RSA
> key, a
> >> known hosts file and the "paraphrase" property descriptor is populated
> >> with
> >> a password. I converted that password over to a parameter within the
> >> master processor group, set the "Process Group Parameter Context" for
> the
> >> processor group this processor resides and saved everything. After that
> >> was done, I was able to access the parameter within #{ + ctrl + space.
> I
> >> choose the correct one, closed in the curly braces and saved the
> >> processor. I noticed immediately after saving and re-opening that the
> new
> >> options icon to the right of the property descriptor reverted back from
> >> "access the parameters" to "convert to a parameter". I didn't think
> much
> >> of it, besides maybe a small bug in the UI (Although that does save
> >> correctly when you set a non sensitive field). When I started the
> >> processor, it immediately failed with an authentication error. I tried
> >> setting a few different parameters, manually typing in #{ftp.password}
> >> into
> >> the paraphrase and even setting a new parameter directly from the helper
> >> icon within the processor. That didn't seem to change the behavior.
> >> Lastly I went back into the processor and removed the "Sensitive value
> >> set" and typed the password in plain text and saved. It worked fine
> after
> >> that.
> >>
> >> I thought it may have been something off with our policies or even
> >> install,
> >> so I installed Nifi locally and experienced the same issue. It seems as
> >> though the actual value is not being evaluated correctly when passed
> >> into/through the processor for sensitive values.
> >>
> >> Has anyone experienced this before?
> >>
> >> Thanks!
> >>
> >> Chris Lundeberg
> >>
> >
> >
> > --
> > -------------------------------
> > Rob Fellows
> >
>
>
> --
> -------------------------------
> Rob Fellows
>
Re: Parameters within password/passphrase fields
Posted by Robert Fellows <ro...@gmail.com>.
Chris,
I was able to reproduce your issue so I just logged the issue. You can
track it here: https://issues.apache.org/jira/browse/NIFI-6868. Feel free
to add any more context/comments as you like.
- Rob
On Wed, Nov 13, 2019 at 3:55 PM Robert Fellows <ro...@gmail.com>
wrote:
> Chris,
> I think there are 2 things happening here. First, I think there might be
> a bug in how sensitive parameters are evaluated on the backend and it is
> causing your issue. Please log an issue for this.
>
> Second, the UI is a bit confusing in terms of the availability of the
> "Convert to parameter" and "Go to Parameter" actions for sensitive
> parameter references in properties. However, this is due to the nature of
> sensitive parameters/properties. The actual value of the sensitive property
> is only ever known to the UI when the user enters the value. Once applied,
> those values are never sent back to the UI. The UI knows there is a value
> set, but it has no idea if it contains a parameter reference or not. The
> "Go To Parameter" action is only available when the UI can parse the value
> of the property and it matches the '#{...}' syntax format. Since the values
> is not sent to the UI, it can't parse the value to know if it is a
> parameter reference or not. The "Convert to parameter" action option is
> available just as a shortcut to create a new parameter, it is hidden when
> it is known that the value already references a parameter. This may be
> improved in the future, but it was implemented this way intentionally.
>
> - Rob
>
> On Wed, Nov 13, 2019 at 11:17 AM Chris Lundeberg <cl...@1904labs.com>
> wrote:
>
>> Hi all,
>>
>> Recently we upgraded to Nifi 1.10 to take advantage of parameters and
>> using
>> those within our deployment process (using the registry). I am seeing
>> some
>> strange behavior and want to confirm my understanding of how this new
>> feature works, before digging deeper.
>>
>> *Some general background info:*
>> I have a three node cluster running Nifi 1.10. Its security enabled, but
>> I
>> am using an admin user with full rights to view/modify policies. Its
>> running on linux and using java 8.
>>
>> *Producing the issue:*
>> As a test, I moved over some existing flows from our 1.9.2 cluster (same
>> specs and user rights as we build everything with puppet). A few of the
>> processors are pulling and pushing to SFTP, for which we use an RSA key, a
>> known hosts file and the "paraphrase" property descriptor is populated
>> with
>> a password. I converted that password over to a parameter within the
>> master processor group, set the "Process Group Parameter Context" for the
>> processor group this processor resides and saved everything. After that
>> was done, I was able to access the parameter within #{ + ctrl + space. I
>> choose the correct one, closed in the curly braces and saved the
>> processor. I noticed immediately after saving and re-opening that the new
>> options icon to the right of the property descriptor reverted back from
>> "access the parameters" to "convert to a parameter". I didn't think much
>> of it, besides maybe a small bug in the UI (Although that does save
>> correctly when you set a non sensitive field). When I started the
>> processor, it immediately failed with an authentication error. I tried
>> setting a few different parameters, manually typing in #{ftp.password}
>> into
>> the paraphrase and even setting a new parameter directly from the helper
>> icon within the processor. That didn't seem to change the behavior.
>> Lastly I went back into the processor and removed the "Sensitive value
>> set" and typed the password in plain text and saved. It worked fine after
>> that.
>>
>> I thought it may have been something off with our policies or even
>> install,
>> so I installed Nifi locally and experienced the same issue. It seems as
>> though the actual value is not being evaluated correctly when passed
>> into/through the processor for sensitive values.
>>
>> Has anyone experienced this before?
>>
>> Thanks!
>>
>> Chris Lundeberg
>>
>
>
> --
> -------------------------------
> Rob Fellows
>
--
-------------------------------
Rob Fellows
Re: Parameters within password/passphrase fields
Posted by Robert Fellows <ro...@gmail.com>.
Chris,
I think there are 2 things happening here. First, I think there might be
a bug in how sensitive parameters are evaluated on the backend and it is
causing your issue. Please log an issue for this.
Second, the UI is a bit confusing in terms of the availability of the
"Convert to parameter" and "Go to Parameter" actions for sensitive
parameter references in properties. However, this is due to the nature of
sensitive parameters/properties. The actual value of the sensitive property
is only ever known to the UI when the user enters the value. Once applied,
those values are never sent back to the UI. The UI knows there is a value
set, but it has no idea if it contains a parameter reference or not. The
"Go To Parameter" action is only available when the UI can parse the value
of the property and it matches the '#{...}' syntax format. Since the values
is not sent to the UI, it can't parse the value to know if it is a
parameter reference or not. The "Convert to parameter" action option is
available just as a shortcut to create a new parameter, it is hidden when
it is known that the value already references a parameter. This may be
improved in the future, but it was implemented this way intentionally.
- Rob
On Wed, Nov 13, 2019 at 11:17 AM Chris Lundeberg <cl...@1904labs.com>
wrote:
> Hi all,
>
> Recently we upgraded to Nifi 1.10 to take advantage of parameters and using
> those within our deployment process (using the registry). I am seeing some
> strange behavior and want to confirm my understanding of how this new
> feature works, before digging deeper.
>
> *Some general background info:*
> I have a three node cluster running Nifi 1.10. Its security enabled, but I
> am using an admin user with full rights to view/modify policies. Its
> running on linux and using java 8.
>
> *Producing the issue:*
> As a test, I moved over some existing flows from our 1.9.2 cluster (same
> specs and user rights as we build everything with puppet). A few of the
> processors are pulling and pushing to SFTP, for which we use an RSA key, a
> known hosts file and the "paraphrase" property descriptor is populated with
> a password. I converted that password over to a parameter within the
> master processor group, set the "Process Group Parameter Context" for the
> processor group this processor resides and saved everything. After that
> was done, I was able to access the parameter within #{ + ctrl + space. I
> choose the correct one, closed in the curly braces and saved the
> processor. I noticed immediately after saving and re-opening that the new
> options icon to the right of the property descriptor reverted back from
> "access the parameters" to "convert to a parameter". I didn't think much
> of it, besides maybe a small bug in the UI (Although that does save
> correctly when you set a non sensitive field). When I started the
> processor, it immediately failed with an authentication error. I tried
> setting a few different parameters, manually typing in #{ftp.password} into
> the paraphrase and even setting a new parameter directly from the helper
> icon within the processor. That didn't seem to change the behavior.
> Lastly I went back into the processor and removed the "Sensitive value
> set" and typed the password in plain text and saved. It worked fine after
> that.
>
> I thought it may have been something off with our policies or even install,
> so I installed Nifi locally and experienced the same issue. It seems as
> though the actual value is not being evaluated correctly when passed
> into/through the processor for sensitive values.
>
> Has anyone experienced this before?
>
> Thanks!
>
> Chris Lundeberg
>
--
-------------------------------
Rob Fellows