You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@nifi.apache.org by Sivaprasanna <si...@gmail.com> on 2018/04/25 16:24:30 UTC
Support for accessing sensitive values safely
Hi
Since flowfile attributes and VariableRegistry is not suitable (not safe,
to be specific), developers have to rely on manually configuring the
sensitive values on the components (Processors & ControllerServices). And
during CI/CD (using flow registry), the sensitive information are dropped
and once imported to the next environment (QA or Prod), the user is
expected to configure the sensitive information again, although for the
first time. How about we introduce sort of a 'vault' that holds sensitive
values which could possibly avoid this unnecessary step completely ?
-
Sivaprasanna
Re: [DISCUSS] Support for accessing sensitive values safely
Posted by Sivaprasanna <si...@gmail.com>.
Andy, That's exactly what I had in mind, but without a separate textbox
popping up. What I had originally though, there is a checkbox instead of a
button named "Use Variables". If it is checked, whatever the user types,
AJAX comes in to play and suggests ( auto-completion ) variables that the
user has access to. If the box is not checked, it is going to be a literal
password. But in this case, the normal textbox that pops up when a property
is edited, because it is more of a "textarea" that a textbox and the
suggestion dropdown would feel awkward there. I haven't worked extensively
on UI so not sure about how difficult it is going to be. Definitely UI
folks would pitch in much better ideas.
Thanks,
Sivaprasanna
On Thu, Apr 26, 2018 at 7:25 PM, Andy LoPresto <al...@apache.org> wrote:
> Yes, I think a dynamically-populated dropdown is unnecessary, and I think
> there might be a way to do this without requiring complex EL parsing.
>
> For sensitive properties, I imagine a text field as we have now, with a
> button to the right that says “Use Variable”. I do not anticipate having to
> support a combination of variable interpolation and literal text (i.e. no
> one’s password should be “${db.password}12345”), so we only have to support
> the use cases of *literal password entry* OR *variable population*.
>
> If the user clicks “Use Variable”, a dialog would appear which has a text
> field and allows the user to type, and uses an AJAX function to call a
> query API endpoint, which allows them to filter the available variables as
> they type. For example, if the user types “d”, the following list might
> appear:
>
> data.endpoint
> db.url
> db.username
> db.password
>
> As the user continues to type “db.p”, only “db.password” is available. The
> user selects this (thus not allowing true freeform input to the final
> value, only a predetermined list of valid values). Obviously this API would
> need to be protected with proper authorization to ensure that only the
> variables available to that user were exposed, but this would be the case
> on validation as well to ensure the user didn’t manually type in a variable
> reference they were not allowed to use.
>
> I’ve included a couple mockups of how I envision this. I’ve implemented
> the same thing in a Rails app a few years ago, so I know it’s possible, and
> with modern libraries, should not be too difficult. This should also
> demonstrate that people like Rob, Matt, and Scott who actually design the
> UI/UX that goes into NiFi are invaluable.
>
> Current sensitive property entry field
> Proposed “Use Variable” button
>
> “Use Variable” dialog with empty text field
>
> User starts typing, triggering AJAX query
>
> User continues typing, narrowing down result set, and selects it
>
> Andy LoPresto
> alopresto@apache.org
> *alopresto.apache@gmail.com <al...@gmail.com>*
> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
>
> On Apr 26, 2018, at 9:05 AM, Bryan Bende <bb...@gmail.com> wrote:
>
> That is a fair point about the list of variables potentially being
> long, we would probably want to get some UI/UX recommendations from
> the folks that have worked in that are the most.
>
> In practice I wonder if it would really be an issue though...
>
> The variables that would be selectable would only be the sensitive
> variables the user has access to, not all variables. I would imagine
> out of hundreds of variables only a handful are sensitive variables.
> It is similar to processor properties in that there are tons of
> properties across all processors, but only a small batch of sensitive
> properties (I just counted 63 references to sensitive(true) in the
> codebase).
>
> The other factor is that it would only be the sensitive variables that
> are in scope to where you are in the data flow, using the same logic
> of how variables are resolved in a hierarchical order. So you would
> only see sensitive variables that are in the current process group or
> a parent group.
>
> As far as knowing if the enter plain password vs. EL, that was the
> reason I suggested selecting a variable from a list. That is the
> mechanism that tells us that a variable was used. The hope was that
> you could enter free form text or select a variable, any time free
> form text was used it would function the way it does today where the
> value is cleared out when saved to registry. If a variable was
> selected then it could keep the variable name.
>
> If we didn't do a list of variables then we would need a checkbox or
> something for the user to indicate whether the sensitive value is a
> literal or expression language.
>
>
> On Thu, Apr 26, 2018 at 5:24 AM, Sivaprasanna <si...@gmail.com>
> wrote:
>
> Initially when I thought of this, I imagined that we can still leverage
> `PropertyDescriptor` and just add new method(s) to handle the sensitive
> variables. Some thought has to be put on this.
>
> *"It would provide a list of variables that are readable to the current
> user and one can be selected"*
> I think this might lead to bad UX. When the number of variables grow,
> rendering the variables in a drop-down list may not be that good, IMHO.
> Maybe we can still stick with textbox and do the following:
>
> - Once a user enters sensitive variable expression language, the
> framework does a background check if the user is authorized to use this
> variable. If the user doesn't have access, render the same and make the
> component invalid there by preventing it to be enabled/started
>
> Only catch is how are we specifying what the user entered is plain password
> and not EL. Either we can go with the assumption that all sensitive values
> have to be taken from Variable Registry and only the EL for those variables
> will have to be provided as the sensitive property value but this would
> restrict the developers' freedom to choose what approach they want. Or we
> can tweak the UI and present the option better, like have a checkbox which
> when checked, will consider that as an EL and evaluate from the variable
> registry and when not checked, assume that the entered password is plain
> password and no evaluation needs to happen.
>
> -
>
> Sivaprasanna
>
> On Thu, Apr 26, 2018 at 12:19 AM, Bryan Bende <bb...@gmail.com> wrote:
>
> The policy model would need more thought, but the point would be that
> a user can select variable references they have been given permission
> to.
>
> In order to configure the processor that is referencing the variable,
> they already need write permissions to that processor, or some parent
> in the hierarchy if no specific policy exists.
>
>
>
> On Wed, Apr 25, 2018 at 2:42 PM, Otto Fowler <ot...@gmail.com>
> wrote:
>
>
> "It would provide a list of variables that are readable to the current
>
> user
>
> and one can be selected, just like allowable values or controller
>
> services.”
>
>
> A person may have rights to configure nifi without knowing the “value” of
> the secure db password ( for example ), but that doesn’t mean they
> don’t have there rights to reference it.
>
>
>
> On April 25, 2018 at 14:15:16, Bryan Bende (bbende@gmail.com) wrote:
>
> There is definitely room for improvement here.
>
> Keep in mind that often the sensitive information is specific to a
> given environment. For example you build a flow in dev with your
> db.password. You don't actually want your dev db password to be
> propagated to the next environment, but you do want to be able to set
> a variable placeholder like ${db.password} and leave that placeholder
> so you can just set that variable in the next environment. So to me
> the goal here is how to handle secure variables.
>
> Andy highlighted many of the issues, my proposal would be the
>
> following...
>
>
> First, we can introduce a concept of a sensitive variable. This would
> be something in the UI where a user can indicate a variable is
> sensitive, maybe a checkbox, and then the framework can store these
> values encrypted (currently all variable values are stored in plain
> text because they aren't meant to be sensitive).
>
> Second, we can introduce policies on sensitive variables so that we
> can restrict who can read them elsewhere, just like policies on
> controller services that determine which controller services show up
> in the drop down of a processor.
>
> Third, we introduce a new kind of PropertyDescriptor that allows
> selecting a variable from the variable registry rather than free-form
> expression language. It would provide a list of variables that are
> readable to the current user and one can be selected, just like
> allowable values or controller services. Ideally we can have a way to
> still allow free form values for people who don't want to use
> variables.
>
> Fourth, anytime variables are evaluated from expression language we
> would prevent evaluating any of these new sensitive variables since we
> have no way of knowing if a user should have access to it from
> free-form EL, so they can only be used from the special
> PropertyDescriptors above.
>
> If we put all this in place then when we save flows to the registry,
> we can leave the variable place-holders in the sensitive properties,
> and then when you import to the next environment you only need to edit
> the variables section and not go through individual processors setting
> sensitive properties.
>
> On Wed, Apr 25, 2018 at 1:06 PM, Andy LoPresto <al...@apache.org>
>
> wrote:
>
> Hi Sivaprasanna,
>
> This was a topic that was briefly considered earlier in the lifecycle of
> the
> project, but was sidelined due to other developments. With the NiFi
> Registry
> project, there has been renewed interest in securing sensitive values in
> the
> flow and allowing for easier import/export/persistence. There is a
> placeholder Jira [1] which doesn’t capture significant information about
> the
> problem. I think a larger conversation needs to occur which covers the
> following points (at a minimum, there is plenty of room for additional
> concerns and use cases):
>
> * How the sensitive values are secured (encryption, storage [HSM [2],
> Hashicorp Vault [3], Square KeyWhiz [4], JCEKS, locally-encrypted file],
> location)
> * User access control (granularity, integration with UAC policies in
>
> NiFi,
>
> Ranger, users/groups, etc.)
> * Exporting/persistence behavior (should a sensitive value entered in
> “dev”
> be exported to “prod” (and more significantly, vice-versa), which
> instance(s) of the Variable Registry are allowed to be referenced from
> each
> NiFi / Registry node, etc.)
> * Variable references (how does the tool differentiate between
> “${db.password}” meaning “load the variable db.password” and a literal
> password like “myPass${word!&”?
>
> The original Jira for encrypted configuration files / properties [5]
>
> also
>
> referenced some of these concepts in the abstract, and there is a rough
> security roadmap in the wiki [6]. The Variable Registry design document
> [7]
> specifically did not allow for sensitive values to be exposed via UI or
> API.
>
> I think there is an appetite for a more complete solution to this
>
> problem
>
> as
> you outlined, but I think there needs to be an extensive collection of
> actual use cases, user expectations, and then technical discussion on
>
> the
>
> implementation to solve this successfully. It’s a minefield where
> half-steps
> will lead to user confusion, unmet expectations, and potentially severe
> security vulnerabilities.
>
> I changed the subject line to include [DISCUSS] to hopefully generate
>
> some
>
> more interest here for other community members to weigh in. Thanks for
> getting the conversation started.
>
> [1] https://issues.apache.org/jira/browse/NIFI-2653
> [2] https://en.wikipedia.org/wiki/Hardware_security_module
> [3] https://www.vaultproject.io/
> [4] https://square.github.io/keywhiz/
> [5] https://issues.apache.org/jira/browse/NIFI-1831
> [6]
> https://cwiki.apache.org/confluence/display/NIFI/
>
> Security+Feature+Roadmap
>
> [7] https://cwiki.apache.org/confluence/display/NIFI/Variable+Registry
>
> Andy LoPresto
> alopresto@apache.org
> alopresto.apache@gmail.com
> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
>
> On Apr 25, 2018, at 12:24 PM, Sivaprasanna <si...@gmail.com>
> wrote:
>
> Hi
>
> Since flowfile attributes and VariableRegistry is not suitable (not
>
> safe,
>
> to be specific), developers have to rely on manually configuring the
> sensitive values on the components (Processors & ControllerServices).
>
> And
>
> during CI/CD (using flow registry), the sensitive information are
>
> dropped
>
> and once imported to the next environment (QA or Prod), the user is
> expected to configure the sensitive information again, although for the
> first time. How about we introduce sort of a 'vault' that holds
>
> sensitive
>
> values which could possibly avoid this unnecessary step completely ?
>
> -
> Sivaprasanna
>
>
>
>
>
Re: [DISCUSS] Support for accessing sensitive values safely
Posted by Andy LoPresto <al...@apache.org>.
Yes, I think a dynamically-populated dropdown is unnecessary, and I think there might be a way to do this without requiring complex EL parsing.
For sensitive properties, I imagine a text field as we have now, with a button to the right that says “Use Variable”. I do not anticipate having to support a combination of variable interpolation and literal text (i.e. no one’s password should be “${db.password}12345”), so we only have to support the use cases of *literal password entry* OR *variable population*.
If the user clicks “Use Variable”, a dialog would appear which has a text field and allows the user to type, and uses an AJAX function to call a query API endpoint, which allows them to filter the available variables as they type. For example, if the user types “d”, the following list might appear:
data.endpoint
db.url
db.username
db.password
As the user continues to type “db.p”, only “db.password” is available. The user selects this (thus not allowing true freeform input to the final value, only a predetermined list of valid values). Obviously this API would need to be protected with proper authorization to ensure that only the variables available to that user were exposed, but this would be the case on validation as well to ensure the user didn’t manually type in a variable reference they were not allowed to use.
I’ve included a couple mockups of how I envision this. I’ve implemented the same thing in a Rails app a few years ago, so I know it’s possible, and with modern libraries, should not be too difficult. This should also demonstrate that people like Rob, Matt, and Scott who actually design the UI/UX that goes into NiFi are invaluable.
Current sensitive property entry field
Proposed “Use Variable” button
“Use Variable” dialog with empty text field
User starts typing, triggering AJAX query
User continues typing, narrowing down result set, and selects it
Andy LoPresto
alopresto@apache.org
alopresto.apache@gmail.com
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
> On Apr 26, 2018, at 9:05 AM, Bryan Bende <bb...@gmail.com> wrote:
>
> That is a fair point about the list of variables potentially being
> long, we would probably want to get some UI/UX recommendations from
> the folks that have worked in that are the most.
>
> In practice I wonder if it would really be an issue though...
>
> The variables that would be selectable would only be the sensitive
> variables the user has access to, not all variables. I would imagine
> out of hundreds of variables only a handful are sensitive variables.
> It is similar to processor properties in that there are tons of
> properties across all processors, but only a small batch of sensitive
> properties (I just counted 63 references to sensitive(true) in the
> codebase).
>
> The other factor is that it would only be the sensitive variables that
> are in scope to where you are in the data flow, using the same logic
> of how variables are resolved in a hierarchical order. So you would
> only see sensitive variables that are in the current process group or
> a parent group.
>
> As far as knowing if the enter plain password vs. EL, that was the
> reason I suggested selecting a variable from a list. That is the
> mechanism that tells us that a variable was used. The hope was that
> you could enter free form text or select a variable, any time free
> form text was used it would function the way it does today where the
> value is cleared out when saved to registry. If a variable was
> selected then it could keep the variable name.
>
> If we didn't do a list of variables then we would need a checkbox or
> something for the user to indicate whether the sensitive value is a
> literal or expression language.
>
>
> On Thu, Apr 26, 2018 at 5:24 AM, Sivaprasanna <si...@gmail.com> wrote:
>> Initially when I thought of this, I imagined that we can still leverage
>> `PropertyDescriptor` and just add new method(s) to handle the sensitive
>> variables. Some thought has to be put on this.
>>
>> *"It would provide a list of variables that are readable to the current
>> user and one can be selected"*
>> I think this might lead to bad UX. When the number of variables grow,
>> rendering the variables in a drop-down list may not be that good, IMHO.
>> Maybe we can still stick with textbox and do the following:
>>
>> - Once a user enters sensitive variable expression language, the
>> framework does a background check if the user is authorized to use this
>> variable. If the user doesn't have access, render the same and make the
>> component invalid there by preventing it to be enabled/started
>>
>> Only catch is how are we specifying what the user entered is plain password
>> and not EL. Either we can go with the assumption that all sensitive values
>> have to be taken from Variable Registry and only the EL for those variables
>> will have to be provided as the sensitive property value but this would
>> restrict the developers' freedom to choose what approach they want. Or we
>> can tweak the UI and present the option better, like have a checkbox which
>> when checked, will consider that as an EL and evaluate from the variable
>> registry and when not checked, assume that the entered password is plain
>> password and no evaluation needs to happen.
>>
>> -
>>
>> Sivaprasanna
>>
>> On Thu, Apr 26, 2018 at 12:19 AM, Bryan Bende <bb...@gmail.com> wrote:
>>
>>> The policy model would need more thought, but the point would be that
>>> a user can select variable references they have been given permission
>>> to.
>>>
>>> In order to configure the processor that is referencing the variable,
>>> they already need write permissions to that processor, or some parent
>>> in the hierarchy if no specific policy exists.
>>>
>>>
>>>
>>> On Wed, Apr 25, 2018 at 2:42 PM, Otto Fowler <ot...@gmail.com>
>>> wrote:
>>>>
>>>> "It would provide a list of variables that are readable to the current
>>> user
>>>> and one can be selected, just like allowable values or controller
>>> services.”
>>>>
>>>> A person may have rights to configure nifi without knowing the “value” of
>>>> the secure db password ( for example ), but that doesn’t mean they
>>>> don’t have there rights to reference it.
>>>>
>>>>
>>>>
>>>> On April 25, 2018 at 14:15:16, Bryan Bende (bbende@gmail.com) wrote:
>>>>
>>>> There is definitely room for improvement here.
>>>>
>>>> Keep in mind that often the sensitive information is specific to a
>>>> given environment. For example you build a flow in dev with your
>>>> db.password. You don't actually want your dev db password to be
>>>> propagated to the next environment, but you do want to be able to set
>>>> a variable placeholder like ${db.password} and leave that placeholder
>>>> so you can just set that variable in the next environment. So to me
>>>> the goal here is how to handle secure variables.
>>>>
>>>> Andy highlighted many of the issues, my proposal would be the
>>> following...
>>>>
>>>> First, we can introduce a concept of a sensitive variable. This would
>>>> be something in the UI where a user can indicate a variable is
>>>> sensitive, maybe a checkbox, and then the framework can store these
>>>> values encrypted (currently all variable values are stored in plain
>>>> text because they aren't meant to be sensitive).
>>>>
>>>> Second, we can introduce policies on sensitive variables so that we
>>>> can restrict who can read them elsewhere, just like policies on
>>>> controller services that determine which controller services show up
>>>> in the drop down of a processor.
>>>>
>>>> Third, we introduce a new kind of PropertyDescriptor that allows
>>>> selecting a variable from the variable registry rather than free-form
>>>> expression language. It would provide a list of variables that are
>>>> readable to the current user and one can be selected, just like
>>>> allowable values or controller services. Ideally we can have a way to
>>>> still allow free form values for people who don't want to use
>>>> variables.
>>>>
>>>> Fourth, anytime variables are evaluated from expression language we
>>>> would prevent evaluating any of these new sensitive variables since we
>>>> have no way of knowing if a user should have access to it from
>>>> free-form EL, so they can only be used from the special
>>>> PropertyDescriptors above.
>>>>
>>>> If we put all this in place then when we save flows to the registry,
>>>> we can leave the variable place-holders in the sensitive properties,
>>>> and then when you import to the next environment you only need to edit
>>>> the variables section and not go through individual processors setting
>>>> sensitive properties.
>>>>
>>>> On Wed, Apr 25, 2018 at 1:06 PM, Andy LoPresto <al...@apache.org>
>>> wrote:
>>>>> Hi Sivaprasanna,
>>>>>
>>>>> This was a topic that was briefly considered earlier in the lifecycle of
>>>>> the
>>>>> project, but was sidelined due to other developments. With the NiFi
>>>>> Registry
>>>>> project, there has been renewed interest in securing sensitive values in
>>>>> the
>>>>> flow and allowing for easier import/export/persistence. There is a
>>>>> placeholder Jira [1] which doesn’t capture significant information about
>>>>> the
>>>>> problem. I think a larger conversation needs to occur which covers the
>>>>> following points (at a minimum, there is plenty of room for additional
>>>>> concerns and use cases):
>>>>>
>>>>> * How the sensitive values are secured (encryption, storage [HSM [2],
>>>>> Hashicorp Vault [3], Square KeyWhiz [4], JCEKS, locally-encrypted file],
>>>>> location)
>>>>> * User access control (granularity, integration with UAC policies in
>>> NiFi,
>>>>> Ranger, users/groups, etc.)
>>>>> * Exporting/persistence behavior (should a sensitive value entered in
>>>>> “dev”
>>>>> be exported to “prod” (and more significantly, vice-versa), which
>>>>> instance(s) of the Variable Registry are allowed to be referenced from
>>>>> each
>>>>> NiFi / Registry node, etc.)
>>>>> * Variable references (how does the tool differentiate between
>>>>> “${db.password}” meaning “load the variable db.password” and a literal
>>>>> password like “myPass${word!&”?
>>>>>
>>>>> The original Jira for encrypted configuration files / properties [5]
>>> also
>>>>> referenced some of these concepts in the abstract, and there is a rough
>>>>> security roadmap in the wiki [6]. The Variable Registry design document
>>>>> [7]
>>>>> specifically did not allow for sensitive values to be exposed via UI or
>>>>> API.
>>>>>
>>>>> I think there is an appetite for a more complete solution to this
>>> problem
>>>>> as
>>>>> you outlined, but I think there needs to be an extensive collection of
>>>>> actual use cases, user expectations, and then technical discussion on
>>> the
>>>>> implementation to solve this successfully. It’s a minefield where
>>>>> half-steps
>>>>> will lead to user confusion, unmet expectations, and potentially severe
>>>>> security vulnerabilities.
>>>>>
>>>>> I changed the subject line to include [DISCUSS] to hopefully generate
>>> some
>>>>> more interest here for other community members to weigh in. Thanks for
>>>>> getting the conversation started.
>>>>>
>>>>> [1] https://issues.apache.org/jira/browse/NIFI-2653
>>>>> [2] https://en.wikipedia.org/wiki/Hardware_security_module
>>>>> [3] https://www.vaultproject.io/
>>>>> [4] https://square.github.io/keywhiz/
>>>>> [5] https://issues.apache.org/jira/browse/NIFI-1831
>>>>> [6]
>>>>> https://cwiki.apache.org/confluence/display/NIFI/
>>> Security+Feature+Roadmap
>>>>> [7] https://cwiki.apache.org/confluence/display/NIFI/Variable+Registry
>>>>>
>>>>> Andy LoPresto
>>>>> alopresto@apache.org
>>>>> alopresto.apache@gmail.com
>>>>> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
>>>>>
>>>>> On Apr 25, 2018, at 12:24 PM, Sivaprasanna <si...@gmail.com>
>>>>> wrote:
>>>>>
>>>>> Hi
>>>>>
>>>>> Since flowfile attributes and VariableRegistry is not suitable (not
>>> safe,
>>>>> to be specific), developers have to rely on manually configuring the
>>>>> sensitive values on the components (Processors & ControllerServices).
>>> And
>>>>> during CI/CD (using flow registry), the sensitive information are
>>> dropped
>>>>> and once imported to the next environment (QA or Prod), the user is
>>>>> expected to configure the sensitive information again, although for the
>>>>> first time. How about we introduce sort of a 'vault' that holds
>>> sensitive
>>>>> values which could possibly avoid this unnecessary step completely ?
>>>>>
>>>>> -
>>>>> Sivaprasanna
>>>>>
>>>>>
>>>
Re: [DISCUSS] Support for accessing sensitive values safely
Posted by Bryan Bende <bb...@gmail.com>.
That is a fair point about the list of variables potentially being
long, we would probably want to get some UI/UX recommendations from
the folks that have worked in that are the most.
In practice I wonder if it would really be an issue though...
The variables that would be selectable would only be the sensitive
variables the user has access to, not all variables. I would imagine
out of hundreds of variables only a handful are sensitive variables.
It is similar to processor properties in that there are tons of
properties across all processors, but only a small batch of sensitive
properties (I just counted 63 references to sensitive(true) in the
codebase).
The other factor is that it would only be the sensitive variables that
are in scope to where you are in the data flow, using the same logic
of how variables are resolved in a hierarchical order. So you would
only see sensitive variables that are in the current process group or
a parent group.
As far as knowing if the enter plain password vs. EL, that was the
reason I suggested selecting a variable from a list. That is the
mechanism that tells us that a variable was used. The hope was that
you could enter free form text or select a variable, any time free
form text was used it would function the way it does today where the
value is cleared out when saved to registry. If a variable was
selected then it could keep the variable name.
If we didn't do a list of variables then we would need a checkbox or
something for the user to indicate whether the sensitive value is a
literal or expression language.
On Thu, Apr 26, 2018 at 5:24 AM, Sivaprasanna <si...@gmail.com> wrote:
> Initially when I thought of this, I imagined that we can still leverage
> `PropertyDescriptor` and just add new method(s) to handle the sensitive
> variables. Some thought has to be put on this.
>
> *"It would provide a list of variables that are readable to the current
> user and one can be selected"*
> I think this might lead to bad UX. When the number of variables grow,
> rendering the variables in a drop-down list may not be that good, IMHO.
> Maybe we can still stick with textbox and do the following:
>
> - Once a user enters sensitive variable expression language, the
> framework does a background check if the user is authorized to use this
> variable. If the user doesn't have access, render the same and make the
> component invalid there by preventing it to be enabled/started
>
> Only catch is how are we specifying what the user entered is plain password
> and not EL. Either we can go with the assumption that all sensitive values
> have to be taken from Variable Registry and only the EL for those variables
> will have to be provided as the sensitive property value but this would
> restrict the developers' freedom to choose what approach they want. Or we
> can tweak the UI and present the option better, like have a checkbox which
> when checked, will consider that as an EL and evaluate from the variable
> registry and when not checked, assume that the entered password is plain
> password and no evaluation needs to happen.
>
> -
>
> Sivaprasanna
>
> On Thu, Apr 26, 2018 at 12:19 AM, Bryan Bende <bb...@gmail.com> wrote:
>
>> The policy model would need more thought, but the point would be that
>> a user can select variable references they have been given permission
>> to.
>>
>> In order to configure the processor that is referencing the variable,
>> they already need write permissions to that processor, or some parent
>> in the hierarchy if no specific policy exists.
>>
>>
>>
>> On Wed, Apr 25, 2018 at 2:42 PM, Otto Fowler <ot...@gmail.com>
>> wrote:
>> >
>> > "It would provide a list of variables that are readable to the current
>> user
>> > and one can be selected, just like allowable values or controller
>> services.”
>> >
>> > A person may have rights to configure nifi without knowing the “value” of
>> > the secure db password ( for example ), but that doesn’t mean they
>> > don’t have there rights to reference it.
>> >
>> >
>> >
>> > On April 25, 2018 at 14:15:16, Bryan Bende (bbende@gmail.com) wrote:
>> >
>> > There is definitely room for improvement here.
>> >
>> > Keep in mind that often the sensitive information is specific to a
>> > given environment. For example you build a flow in dev with your
>> > db.password. You don't actually want your dev db password to be
>> > propagated to the next environment, but you do want to be able to set
>> > a variable placeholder like ${db.password} and leave that placeholder
>> > so you can just set that variable in the next environment. So to me
>> > the goal here is how to handle secure variables.
>> >
>> > Andy highlighted many of the issues, my proposal would be the
>> following...
>> >
>> > First, we can introduce a concept of a sensitive variable. This would
>> > be something in the UI where a user can indicate a variable is
>> > sensitive, maybe a checkbox, and then the framework can store these
>> > values encrypted (currently all variable values are stored in plain
>> > text because they aren't meant to be sensitive).
>> >
>> > Second, we can introduce policies on sensitive variables so that we
>> > can restrict who can read them elsewhere, just like policies on
>> > controller services that determine which controller services show up
>> > in the drop down of a processor.
>> >
>> > Third, we introduce a new kind of PropertyDescriptor that allows
>> > selecting a variable from the variable registry rather than free-form
>> > expression language. It would provide a list of variables that are
>> > readable to the current user and one can be selected, just like
>> > allowable values or controller services. Ideally we can have a way to
>> > still allow free form values for people who don't want to use
>> > variables.
>> >
>> > Fourth, anytime variables are evaluated from expression language we
>> > would prevent evaluating any of these new sensitive variables since we
>> > have no way of knowing if a user should have access to it from
>> > free-form EL, so they can only be used from the special
>> > PropertyDescriptors above.
>> >
>> > If we put all this in place then when we save flows to the registry,
>> > we can leave the variable place-holders in the sensitive properties,
>> > and then when you import to the next environment you only need to edit
>> > the variables section and not go through individual processors setting
>> > sensitive properties.
>> >
>> > On Wed, Apr 25, 2018 at 1:06 PM, Andy LoPresto <al...@apache.org>
>> wrote:
>> >> Hi Sivaprasanna,
>> >>
>> >> This was a topic that was briefly considered earlier in the lifecycle of
>> >> the
>> >> project, but was sidelined due to other developments. With the NiFi
>> >> Registry
>> >> project, there has been renewed interest in securing sensitive values in
>> >> the
>> >> flow and allowing for easier import/export/persistence. There is a
>> >> placeholder Jira [1] which doesn’t capture significant information about
>> >> the
>> >> problem. I think a larger conversation needs to occur which covers the
>> >> following points (at a minimum, there is plenty of room for additional
>> >> concerns and use cases):
>> >>
>> >> * How the sensitive values are secured (encryption, storage [HSM [2],
>> >> Hashicorp Vault [3], Square KeyWhiz [4], JCEKS, locally-encrypted file],
>> >> location)
>> >> * User access control (granularity, integration with UAC policies in
>> NiFi,
>> >> Ranger, users/groups, etc.)
>> >> * Exporting/persistence behavior (should a sensitive value entered in
>> >> “dev”
>> >> be exported to “prod” (and more significantly, vice-versa), which
>> >> instance(s) of the Variable Registry are allowed to be referenced from
>> >> each
>> >> NiFi / Registry node, etc.)
>> >> * Variable references (how does the tool differentiate between
>> >> “${db.password}” meaning “load the variable db.password” and a literal
>> >> password like “myPass${word!&”?
>> >>
>> >> The original Jira for encrypted configuration files / properties [5]
>> also
>> >> referenced some of these concepts in the abstract, and there is a rough
>> >> security roadmap in the wiki [6]. The Variable Registry design document
>> >> [7]
>> >> specifically did not allow for sensitive values to be exposed via UI or
>> >> API.
>> >>
>> >> I think there is an appetite for a more complete solution to this
>> problem
>> >> as
>> >> you outlined, but I think there needs to be an extensive collection of
>> >> actual use cases, user expectations, and then technical discussion on
>> the
>> >> implementation to solve this successfully. It’s a minefield where
>> >> half-steps
>> >> will lead to user confusion, unmet expectations, and potentially severe
>> >> security vulnerabilities.
>> >>
>> >> I changed the subject line to include [DISCUSS] to hopefully generate
>> some
>> >> more interest here for other community members to weigh in. Thanks for
>> >> getting the conversation started.
>> >>
>> >> [1] https://issues.apache.org/jira/browse/NIFI-2653
>> >> [2] https://en.wikipedia.org/wiki/Hardware_security_module
>> >> [3] https://www.vaultproject.io/
>> >> [4] https://square.github.io/keywhiz/
>> >> [5] https://issues.apache.org/jira/browse/NIFI-1831
>> >> [6]
>> >> https://cwiki.apache.org/confluence/display/NIFI/
>> Security+Feature+Roadmap
>> >> [7] https://cwiki.apache.org/confluence/display/NIFI/Variable+Registry
>> >>
>> >> Andy LoPresto
>> >> alopresto@apache.org
>> >> alopresto.apache@gmail.com
>> >> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
>> >>
>> >> On Apr 25, 2018, at 12:24 PM, Sivaprasanna <si...@gmail.com>
>> >> wrote:
>> >>
>> >> Hi
>> >>
>> >> Since flowfile attributes and VariableRegistry is not suitable (not
>> safe,
>> >> to be specific), developers have to rely on manually configuring the
>> >> sensitive values on the components (Processors & ControllerServices).
>> And
>> >> during CI/CD (using flow registry), the sensitive information are
>> dropped
>> >> and once imported to the next environment (QA or Prod), the user is
>> >> expected to configure the sensitive information again, although for the
>> >> first time. How about we introduce sort of a 'vault' that holds
>> sensitive
>> >> values which could possibly avoid this unnecessary step completely ?
>> >>
>> >> -
>> >> Sivaprasanna
>> >>
>> >>
>>
Re: [DISCUSS] Support for accessing sensitive values safely
Posted by Sivaprasanna <si...@gmail.com>.
Initially when I thought of this, I imagined that we can still leverage
`PropertyDescriptor` and just add new method(s) to handle the sensitive
variables. Some thought has to be put on this.
*"It would provide a list of variables that are readable to the current
user and one can be selected"*
I think this might lead to bad UX. When the number of variables grow,
rendering the variables in a drop-down list may not be that good, IMHO.
Maybe we can still stick with textbox and do the following:
- Once a user enters sensitive variable expression language, the
framework does a background check if the user is authorized to use this
variable. If the user doesn't have access, render the same and make the
component invalid there by preventing it to be enabled/started
Only catch is how are we specifying what the user entered is plain password
and not EL. Either we can go with the assumption that all sensitive values
have to be taken from Variable Registry and only the EL for those variables
will have to be provided as the sensitive property value but this would
restrict the developers' freedom to choose what approach they want. Or we
can tweak the UI and present the option better, like have a checkbox which
when checked, will consider that as an EL and evaluate from the variable
registry and when not checked, assume that the entered password is plain
password and no evaluation needs to happen.
-
Sivaprasanna
On Thu, Apr 26, 2018 at 12:19 AM, Bryan Bende <bb...@gmail.com> wrote:
> The policy model would need more thought, but the point would be that
> a user can select variable references they have been given permission
> to.
>
> In order to configure the processor that is referencing the variable,
> they already need write permissions to that processor, or some parent
> in the hierarchy if no specific policy exists.
>
>
>
> On Wed, Apr 25, 2018 at 2:42 PM, Otto Fowler <ot...@gmail.com>
> wrote:
> >
> > "It would provide a list of variables that are readable to the current
> user
> > and one can be selected, just like allowable values or controller
> services.”
> >
> > A person may have rights to configure nifi without knowing the “value” of
> > the secure db password ( for example ), but that doesn’t mean they
> > don’t have there rights to reference it.
> >
> >
> >
> > On April 25, 2018 at 14:15:16, Bryan Bende (bbende@gmail.com) wrote:
> >
> > There is definitely room for improvement here.
> >
> > Keep in mind that often the sensitive information is specific to a
> > given environment. For example you build a flow in dev with your
> > db.password. You don't actually want your dev db password to be
> > propagated to the next environment, but you do want to be able to set
> > a variable placeholder like ${db.password} and leave that placeholder
> > so you can just set that variable in the next environment. So to me
> > the goal here is how to handle secure variables.
> >
> > Andy highlighted many of the issues, my proposal would be the
> following...
> >
> > First, we can introduce a concept of a sensitive variable. This would
> > be something in the UI where a user can indicate a variable is
> > sensitive, maybe a checkbox, and then the framework can store these
> > values encrypted (currently all variable values are stored in plain
> > text because they aren't meant to be sensitive).
> >
> > Second, we can introduce policies on sensitive variables so that we
> > can restrict who can read them elsewhere, just like policies on
> > controller services that determine which controller services show up
> > in the drop down of a processor.
> >
> > Third, we introduce a new kind of PropertyDescriptor that allows
> > selecting a variable from the variable registry rather than free-form
> > expression language. It would provide a list of variables that are
> > readable to the current user and one can be selected, just like
> > allowable values or controller services. Ideally we can have a way to
> > still allow free form values for people who don't want to use
> > variables.
> >
> > Fourth, anytime variables are evaluated from expression language we
> > would prevent evaluating any of these new sensitive variables since we
> > have no way of knowing if a user should have access to it from
> > free-form EL, so they can only be used from the special
> > PropertyDescriptors above.
> >
> > If we put all this in place then when we save flows to the registry,
> > we can leave the variable place-holders in the sensitive properties,
> > and then when you import to the next environment you only need to edit
> > the variables section and not go through individual processors setting
> > sensitive properties.
> >
> > On Wed, Apr 25, 2018 at 1:06 PM, Andy LoPresto <al...@apache.org>
> wrote:
> >> Hi Sivaprasanna,
> >>
> >> This was a topic that was briefly considered earlier in the lifecycle of
> >> the
> >> project, but was sidelined due to other developments. With the NiFi
> >> Registry
> >> project, there has been renewed interest in securing sensitive values in
> >> the
> >> flow and allowing for easier import/export/persistence. There is a
> >> placeholder Jira [1] which doesn’t capture significant information about
> >> the
> >> problem. I think a larger conversation needs to occur which covers the
> >> following points (at a minimum, there is plenty of room for additional
> >> concerns and use cases):
> >>
> >> * How the sensitive values are secured (encryption, storage [HSM [2],
> >> Hashicorp Vault [3], Square KeyWhiz [4], JCEKS, locally-encrypted file],
> >> location)
> >> * User access control (granularity, integration with UAC policies in
> NiFi,
> >> Ranger, users/groups, etc.)
> >> * Exporting/persistence behavior (should a sensitive value entered in
> >> “dev”
> >> be exported to “prod” (and more significantly, vice-versa), which
> >> instance(s) of the Variable Registry are allowed to be referenced from
> >> each
> >> NiFi / Registry node, etc.)
> >> * Variable references (how does the tool differentiate between
> >> “${db.password}” meaning “load the variable db.password” and a literal
> >> password like “myPass${word!&”?
> >>
> >> The original Jira for encrypted configuration files / properties [5]
> also
> >> referenced some of these concepts in the abstract, and there is a rough
> >> security roadmap in the wiki [6]. The Variable Registry design document
> >> [7]
> >> specifically did not allow for sensitive values to be exposed via UI or
> >> API.
> >>
> >> I think there is an appetite for a more complete solution to this
> problem
> >> as
> >> you outlined, but I think there needs to be an extensive collection of
> >> actual use cases, user expectations, and then technical discussion on
> the
> >> implementation to solve this successfully. It’s a minefield where
> >> half-steps
> >> will lead to user confusion, unmet expectations, and potentially severe
> >> security vulnerabilities.
> >>
> >> I changed the subject line to include [DISCUSS] to hopefully generate
> some
> >> more interest here for other community members to weigh in. Thanks for
> >> getting the conversation started.
> >>
> >> [1] https://issues.apache.org/jira/browse/NIFI-2653
> >> [2] https://en.wikipedia.org/wiki/Hardware_security_module
> >> [3] https://www.vaultproject.io/
> >> [4] https://square.github.io/keywhiz/
> >> [5] https://issues.apache.org/jira/browse/NIFI-1831
> >> [6]
> >> https://cwiki.apache.org/confluence/display/NIFI/
> Security+Feature+Roadmap
> >> [7] https://cwiki.apache.org/confluence/display/NIFI/Variable+Registry
> >>
> >> Andy LoPresto
> >> alopresto@apache.org
> >> alopresto.apache@gmail.com
> >> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
> >>
> >> On Apr 25, 2018, at 12:24 PM, Sivaprasanna <si...@gmail.com>
> >> wrote:
> >>
> >> Hi
> >>
> >> Since flowfile attributes and VariableRegistry is not suitable (not
> safe,
> >> to be specific), developers have to rely on manually configuring the
> >> sensitive values on the components (Processors & ControllerServices).
> And
> >> during CI/CD (using flow registry), the sensitive information are
> dropped
> >> and once imported to the next environment (QA or Prod), the user is
> >> expected to configure the sensitive information again, although for the
> >> first time. How about we introduce sort of a 'vault' that holds
> sensitive
> >> values which could possibly avoid this unnecessary step completely ?
> >>
> >> -
> >> Sivaprasanna
> >>
> >>
>
Re: [DISCUSS] Support for accessing sensitive values safely
Posted by Bryan Bende <bb...@gmail.com>.
The policy model would need more thought, but the point would be that
a user can select variable references they have been given permission
to.
In order to configure the processor that is referencing the variable,
they already need write permissions to that processor, or some parent
in the hierarchy if no specific policy exists.
On Wed, Apr 25, 2018 at 2:42 PM, Otto Fowler <ot...@gmail.com> wrote:
>
> "It would provide a list of variables that are readable to the current user
> and one can be selected, just like allowable values or controller services.”
>
> A person may have rights to configure nifi without knowing the “value” of
> the secure db password ( for example ), but that doesn’t mean they
> don’t have there rights to reference it.
>
>
>
> On April 25, 2018 at 14:15:16, Bryan Bende (bbende@gmail.com) wrote:
>
> There is definitely room for improvement here.
>
> Keep in mind that often the sensitive information is specific to a
> given environment. For example you build a flow in dev with your
> db.password. You don't actually want your dev db password to be
> propagated to the next environment, but you do want to be able to set
> a variable placeholder like ${db.password} and leave that placeholder
> so you can just set that variable in the next environment. So to me
> the goal here is how to handle secure variables.
>
> Andy highlighted many of the issues, my proposal would be the following...
>
> First, we can introduce a concept of a sensitive variable. This would
> be something in the UI where a user can indicate a variable is
> sensitive, maybe a checkbox, and then the framework can store these
> values encrypted (currently all variable values are stored in plain
> text because they aren't meant to be sensitive).
>
> Second, we can introduce policies on sensitive variables so that we
> can restrict who can read them elsewhere, just like policies on
> controller services that determine which controller services show up
> in the drop down of a processor.
>
> Third, we introduce a new kind of PropertyDescriptor that allows
> selecting a variable from the variable registry rather than free-form
> expression language. It would provide a list of variables that are
> readable to the current user and one can be selected, just like
> allowable values or controller services. Ideally we can have a way to
> still allow free form values for people who don't want to use
> variables.
>
> Fourth, anytime variables are evaluated from expression language we
> would prevent evaluating any of these new sensitive variables since we
> have no way of knowing if a user should have access to it from
> free-form EL, so they can only be used from the special
> PropertyDescriptors above.
>
> If we put all this in place then when we save flows to the registry,
> we can leave the variable place-holders in the sensitive properties,
> and then when you import to the next environment you only need to edit
> the variables section and not go through individual processors setting
> sensitive properties.
>
> On Wed, Apr 25, 2018 at 1:06 PM, Andy LoPresto <al...@apache.org> wrote:
>> Hi Sivaprasanna,
>>
>> This was a topic that was briefly considered earlier in the lifecycle of
>> the
>> project, but was sidelined due to other developments. With the NiFi
>> Registry
>> project, there has been renewed interest in securing sensitive values in
>> the
>> flow and allowing for easier import/export/persistence. There is a
>> placeholder Jira [1] which doesn’t capture significant information about
>> the
>> problem. I think a larger conversation needs to occur which covers the
>> following points (at a minimum, there is plenty of room for additional
>> concerns and use cases):
>>
>> * How the sensitive values are secured (encryption, storage [HSM [2],
>> Hashicorp Vault [3], Square KeyWhiz [4], JCEKS, locally-encrypted file],
>> location)
>> * User access control (granularity, integration with UAC policies in NiFi,
>> Ranger, users/groups, etc.)
>> * Exporting/persistence behavior (should a sensitive value entered in
>> “dev”
>> be exported to “prod” (and more significantly, vice-versa), which
>> instance(s) of the Variable Registry are allowed to be referenced from
>> each
>> NiFi / Registry node, etc.)
>> * Variable references (how does the tool differentiate between
>> “${db.password}” meaning “load the variable db.password” and a literal
>> password like “myPass${word!&”?
>>
>> The original Jira for encrypted configuration files / properties [5] also
>> referenced some of these concepts in the abstract, and there is a rough
>> security roadmap in the wiki [6]. The Variable Registry design document
>> [7]
>> specifically did not allow for sensitive values to be exposed via UI or
>> API.
>>
>> I think there is an appetite for a more complete solution to this problem
>> as
>> you outlined, but I think there needs to be an extensive collection of
>> actual use cases, user expectations, and then technical discussion on the
>> implementation to solve this successfully. It’s a minefield where
>> half-steps
>> will lead to user confusion, unmet expectations, and potentially severe
>> security vulnerabilities.
>>
>> I changed the subject line to include [DISCUSS] to hopefully generate some
>> more interest here for other community members to weigh in. Thanks for
>> getting the conversation started.
>>
>> [1] https://issues.apache.org/jira/browse/NIFI-2653
>> [2] https://en.wikipedia.org/wiki/Hardware_security_module
>> [3] https://www.vaultproject.io/
>> [4] https://square.github.io/keywhiz/
>> [5] https://issues.apache.org/jira/browse/NIFI-1831
>> [6]
>> https://cwiki.apache.org/confluence/display/NIFI/Security+Feature+Roadmap
>> [7] https://cwiki.apache.org/confluence/display/NIFI/Variable+Registry
>>
>> Andy LoPresto
>> alopresto@apache.org
>> alopresto.apache@gmail.com
>> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
>>
>> On Apr 25, 2018, at 12:24 PM, Sivaprasanna <si...@gmail.com>
>> wrote:
>>
>> Hi
>>
>> Since flowfile attributes and VariableRegistry is not suitable (not safe,
>> to be specific), developers have to rely on manually configuring the
>> sensitive values on the components (Processors & ControllerServices). And
>> during CI/CD (using flow registry), the sensitive information are dropped
>> and once imported to the next environment (QA or Prod), the user is
>> expected to configure the sensitive information again, although for the
>> first time. How about we introduce sort of a 'vault' that holds sensitive
>> values which could possibly avoid this unnecessary step completely ?
>>
>> -
>> Sivaprasanna
>>
>>
Re: [DISCUSS] Support for accessing sensitive values safely
Posted by Otto Fowler <ot...@gmail.com>.
"It would provide a list of variables that are readable to the current user
and one can be selected, just like allowable values or controller services.”
A person may have rights to configure nifi without knowing the “value” of
the secure db password ( for example ), but that doesn’t mean they
don’t have there rights to reference it.
On April 25, 2018 at 14:15:16, Bryan Bende (bbende@gmail.com) wrote:
There is definitely room for improvement here.
Keep in mind that often the sensitive information is specific to a
given environment. For example you build a flow in dev with your
db.password. You don't actually want your dev db password to be
propagated to the next environment, but you do want to be able to set
a variable placeholder like ${db.password} and leave that placeholder
so you can just set that variable in the next environment. So to me
the goal here is how to handle secure variables.
Andy highlighted many of the issues, my proposal would be the following...
First, we can introduce a concept of a sensitive variable. This would
be something in the UI where a user can indicate a variable is
sensitive, maybe a checkbox, and then the framework can store these
values encrypted (currently all variable values are stored in plain
text because they aren't meant to be sensitive).
Second, we can introduce policies on sensitive variables so that we
can restrict who can read them elsewhere, just like policies on
controller services that determine which controller services show up
in the drop down of a processor.
Third, we introduce a new kind of PropertyDescriptor that allows
selecting a variable from the variable registry rather than free-form
expression language. It would provide a list of variables that are
readable to the current user and one can be selected, just like
allowable values or controller services. Ideally we can have a way to
still allow free form values for people who don't want to use
variables.
Fourth, anytime variables are evaluated from expression language we
would prevent evaluating any of these new sensitive variables since we
have no way of knowing if a user should have access to it from
free-form EL, so they can only be used from the special
PropertyDescriptors above.
If we put all this in place then when we save flows to the registry,
we can leave the variable place-holders in the sensitive properties,
and then when you import to the next environment you only need to edit
the variables section and not go through individual processors setting
sensitive properties.
On Wed, Apr 25, 2018 at 1:06 PM, Andy LoPresto <al...@apache.org>
wrote:
> Hi Sivaprasanna,
>
> This was a topic that was briefly considered earlier in the lifecycle of
the
> project, but was sidelined due to other developments. With the NiFi
Registry
> project, there has been renewed interest in securing sensitive values in
the
> flow and allowing for easier import/export/persistence. There is a
> placeholder Jira [1] which doesn’t capture significant information about
the
> problem. I think a larger conversation needs to occur which covers the
> following points (at a minimum, there is plenty of room for additional
> concerns and use cases):
>
> * How the sensitive values are secured (encryption, storage [HSM [2],
> Hashicorp Vault [3], Square KeyWhiz [4], JCEKS, locally-encrypted file],
> location)
> * User access control (granularity, integration with UAC policies in
NiFi,
> Ranger, users/groups, etc.)
> * Exporting/persistence behavior (should a sensitive value entered in
“dev”
> be exported to “prod” (and more significantly, vice-versa), which
> instance(s) of the Variable Registry are allowed to be referenced from
each
> NiFi / Registry node, etc.)
> * Variable references (how does the tool differentiate between
> “${db.password}” meaning “load the variable db.password” and a literal
> password like “myPass${word!&”?
>
> The original Jira for encrypted configuration files / properties [5] also
> referenced some of these concepts in the abstract, and there is a rough
> security roadmap in the wiki [6]. The Variable Registry design document
[7]
> specifically did not allow for sensitive values to be exposed via UI or
API.
>
> I think there is an appetite for a more complete solution to this problem
as
> you outlined, but I think there needs to be an extensive collection of
> actual use cases, user expectations, and then technical discussion on the
> implementation to solve this successfully. It’s a minefield where
half-steps
> will lead to user confusion, unmet expectations, and potentially severe
> security vulnerabilities.
>
> I changed the subject line to include [DISCUSS] to hopefully generate
some
> more interest here for other community members to weigh in. Thanks for
> getting the conversation started.
>
> [1] https://issues.apache.org/jira/browse/NIFI-2653
> [2] https://en.wikipedia.org/wiki/Hardware_security_module
> [3] https://www.vaultproject.io/
> [4] https://square.github.io/keywhiz/
> [5] https://issues.apache.org/jira/browse/NIFI-1831
> [6]
> https://cwiki.apache.org/confluence/display/NIFI/Security+Feature+Roadmap
> [7] https://cwiki.apache.org/confluence/display/NIFI/Variable+Registry
>
> Andy LoPresto
> alopresto@apache.org
> alopresto.apache@gmail.com
> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
>
> On Apr 25, 2018, at 12:24 PM, Sivaprasanna <si...@gmail.com>
> wrote:
>
> Hi
>
> Since flowfile attributes and VariableRegistry is not suitable (not safe,
> to be specific), developers have to rely on manually configuring the
> sensitive values on the components (Processors & ControllerServices). And
> during CI/CD (using flow registry), the sensitive information are dropped
> and once imported to the next environment (QA or Prod), the user is
> expected to configure the sensitive information again, although for the
> first time. How about we introduce sort of a 'vault' that holds sensitive
> values which could possibly avoid this unnecessary step completely ?
>
> -
> Sivaprasanna
>
>
Re: [DISCUSS] Support for accessing sensitive values safely
Posted by Bryan Bende <bb...@gmail.com>.
There is definitely room for improvement here.
Keep in mind that often the sensitive information is specific to a
given environment. For example you build a flow in dev with your
db.password. You don't actually want your dev db password to be
propagated to the next environment, but you do want to be able to set
a variable placeholder like ${db.password} and leave that placeholder
so you can just set that variable in the next environment. So to me
the goal here is how to handle secure variables.
Andy highlighted many of the issues, my proposal would be the following...
First, we can introduce a concept of a sensitive variable. This would
be something in the UI where a user can indicate a variable is
sensitive, maybe a checkbox, and then the framework can store these
values encrypted (currently all variable values are stored in plain
text because they aren't meant to be sensitive).
Second, we can introduce policies on sensitive variables so that we
can restrict who can read them elsewhere, just like policies on
controller services that determine which controller services show up
in the drop down of a processor.
Third, we introduce a new kind of PropertyDescriptor that allows
selecting a variable from the variable registry rather than free-form
expression language. It would provide a list of variables that are
readable to the current user and one can be selected, just like
allowable values or controller services. Ideally we can have a way to
still allow free form values for people who don't want to use
variables.
Fourth, anytime variables are evaluated from expression language we
would prevent evaluating any of these new sensitive variables since we
have no way of knowing if a user should have access to it from
free-form EL, so they can only be used from the special
PropertyDescriptors above.
If we put all this in place then when we save flows to the registry,
we can leave the variable place-holders in the sensitive properties,
and then when you import to the next environment you only need to edit
the variables section and not go through individual processors setting
sensitive properties.
On Wed, Apr 25, 2018 at 1:06 PM, Andy LoPresto <al...@apache.org> wrote:
> Hi Sivaprasanna,
>
> This was a topic that was briefly considered earlier in the lifecycle of the
> project, but was sidelined due to other developments. With the NiFi Registry
> project, there has been renewed interest in securing sensitive values in the
> flow and allowing for easier import/export/persistence. There is a
> placeholder Jira [1] which doesn’t capture significant information about the
> problem. I think a larger conversation needs to occur which covers the
> following points (at a minimum, there is plenty of room for additional
> concerns and use cases):
>
> * How the sensitive values are secured (encryption, storage [HSM [2],
> Hashicorp Vault [3], Square KeyWhiz [4], JCEKS, locally-encrypted file],
> location)
> * User access control (granularity, integration with UAC policies in NiFi,
> Ranger, users/groups, etc.)
> * Exporting/persistence behavior (should a sensitive value entered in “dev”
> be exported to “prod” (and more significantly, vice-versa), which
> instance(s) of the Variable Registry are allowed to be referenced from each
> NiFi / Registry node, etc.)
> * Variable references (how does the tool differentiate between
> “${db.password}” meaning “load the variable db.password” and a literal
> password like “myPass${word!&”?
>
> The original Jira for encrypted configuration files / properties [5] also
> referenced some of these concepts in the abstract, and there is a rough
> security roadmap in the wiki [6]. The Variable Registry design document [7]
> specifically did not allow for sensitive values to be exposed via UI or API.
>
> I think there is an appetite for a more complete solution to this problem as
> you outlined, but I think there needs to be an extensive collection of
> actual use cases, user expectations, and then technical discussion on the
> implementation to solve this successfully. It’s a minefield where half-steps
> will lead to user confusion, unmet expectations, and potentially severe
> security vulnerabilities.
>
> I changed the subject line to include [DISCUSS] to hopefully generate some
> more interest here for other community members to weigh in. Thanks for
> getting the conversation started.
>
> [1] https://issues.apache.org/jira/browse/NIFI-2653
> [2] https://en.wikipedia.org/wiki/Hardware_security_module
> [3] https://www.vaultproject.io/
> [4] https://square.github.io/keywhiz/
> [5] https://issues.apache.org/jira/browse/NIFI-1831
> [6]
> https://cwiki.apache.org/confluence/display/NIFI/Security+Feature+Roadmap
> [7] https://cwiki.apache.org/confluence/display/NIFI/Variable+Registry
>
> Andy LoPresto
> alopresto@apache.org
> alopresto.apache@gmail.com
> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
>
> On Apr 25, 2018, at 12:24 PM, Sivaprasanna <si...@gmail.com>
> wrote:
>
> Hi
>
> Since flowfile attributes and VariableRegistry is not suitable (not safe,
> to be specific), developers have to rely on manually configuring the
> sensitive values on the components (Processors & ControllerServices). And
> during CI/CD (using flow registry), the sensitive information are dropped
> and once imported to the next environment (QA or Prod), the user is
> expected to configure the sensitive information again, although for the
> first time. How about we introduce sort of a 'vault' that holds sensitive
> values which could possibly avoid this unnecessary step completely ?
>
> -
> Sivaprasanna
>
>
Re: [DISCUSS] Support for accessing sensitive values safely
Posted by Andy LoPresto <al...@apache.org>.
Hi Sivaprasanna,
This was a topic that was briefly considered earlier in the lifecycle of the project, but was sidelined due to other developments. With the NiFi Registry project, there has been renewed interest in securing sensitive values in the flow and allowing for easier import/export/persistence. There is a placeholder Jira [1] which doesn’t capture significant information about the problem. I think a larger conversation needs to occur which covers the following points (at a minimum, there is plenty of room for additional concerns and use cases):
* How the sensitive values are secured (encryption, storage [HSM [2], Hashicorp Vault [3], Square KeyWhiz [4], JCEKS, locally-encrypted file], location)
* User access control (granularity, integration with UAC policies in NiFi, Ranger, users/groups, etc.)
* Exporting/persistence behavior (should a sensitive value entered in “dev” be exported to “prod” (and more significantly, vice-versa), which instance(s) of the Variable Registry are allowed to be referenced from each NiFi / Registry node, etc.)
* Variable references (how does the tool differentiate between “${db.password}” meaning “load the variable db.password” and a literal password like “myPass${word!&”?
The original Jira for encrypted configuration files / properties [5] also referenced some of these concepts in the abstract, and there is a rough security roadmap in the wiki [6]. The Variable Registry design document [7] specifically did not allow for sensitive values to be exposed via UI or API.
I think there is an appetite for a more complete solution to this problem as you outlined, but I think there needs to be an extensive collection of actual use cases, user expectations, and then technical discussion on the implementation to solve this successfully. It’s a minefield where half-steps will lead to user confusion, unmet expectations, and potentially severe security vulnerabilities.
I changed the subject line to include [DISCUSS] to hopefully generate some more interest here for other community members to weigh in. Thanks for getting the conversation started.
[1] https://issues.apache.org/jira/browse/NIFI-2653 <https://issues.apache.org/jira/browse/NIFI-2653>
[2] https://en.wikipedia.org/wiki/Hardware_security_module <https://en.wikipedia.org/wiki/Hardware_security_module>
[3] https://www.vaultproject.io/ <https://www.vaultproject.io/>
[4] https://square.github.io/keywhiz/ <https://square.github.io/keywhiz/>
[5] https://issues.apache.org/jira/browse/NIFI-1831 <https://issues.apache.org/jira/browse/NIFI-1831>
[6] https://cwiki.apache.org/confluence/display/NIFI/Security+Feature+Roadmap <https://cwiki.apache.org/confluence/display/NIFI/Security+Feature+Roadmap>
[7] https://cwiki.apache.org/confluence/display/NIFI/Variable+Registry <https://cwiki.apache.org/confluence/display/NIFI/Variable+Registry>
Andy LoPresto
alopresto@apache.org
alopresto.apache@gmail.com
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
> On Apr 25, 2018, at 12:24 PM, Sivaprasanna <si...@gmail.com> wrote:
>
> Hi
>
> Since flowfile attributes and VariableRegistry is not suitable (not safe,
> to be specific), developers have to rely on manually configuring the
> sensitive values on the components (Processors & ControllerServices). And
> during CI/CD (using flow registry), the sensitive information are dropped
> and once imported to the next environment (QA or Prod), the user is
> expected to configure the sensitive information again, although for the
> first time. How about we introduce sort of a 'vault' that holds sensitive
> values which could possibly avoid this unnecessary step completely ?
>
> -
> Sivaprasanna