dynamic properties not recovering from ghost component state

[Nissim Shiman <ns...@yahoo.com.INVALID>]

Hello apache nifi devs,

I have run into an issue that appears to require a different way of handling ghost components than we are currently, and would like input before proceeding further.


Specifically, this is related to   https://issues.apache.org/jira/browse/NIFI-11570


Besides the example in the ticket, a good case where this issue can be seen is with the ExecuteScript processor with two dynamic properties, one sensitive and one not.  On recovery from ghost component, both properties will be sensitive.


The challenge is that on recovery from ghost component there no way of knowing if a dynamic property is sensitive or not, so it is left in the existing sensitive/encrypted state.


There are solutions, but each is a shift from how nifi is currently working.

1.  Leave unencrypted properties in the clear when entering ghost component mode.  This would work, but we'd lose protections of why encryption was done in the first place, so probably a non-starter.

2.  When encrypting properties in ghost mode, have a custom encryption tag in flow.json.gz/flow.xml.gz.   i.e. instead of sensitive values starting with enc{   they would start with enc-ghost{   This will help when exiting ghost mode where we could now more easily determine why a value was encrypted in the first place, but is introducing new complexities in keeping track of why something is encrypted


3.  Leave handling of ghost components as is, but for a given component only allow Dynamic Properties to be non-sensitive or senstive, but not both.  This would affect ExecuteScript, InvokeHTTP (and others) which currently allow both and would be rolling back considerable work that was done to allow this, and we'd lose existing functionality, so also not likely.


4.  Leave as is.  It is not so common to have components in ghost state, and only some of them allow Dynamic Properties so this is an acceptable risk considering the alternatives.

I am interested in community feedback and/or other suggestions to the current situation. 


Thanks,


Nissim Shiman

Re: dynamic properties not recovering from ghost component state

[Bryan Bende <bb...@gmail.com>]

Hi Nissim,

I'm wondering if we need to store the set of sensitiveDynamicPropertyNames
in flow.json or flow.xml. Currently when loading the flow we are recreating
this set by looking at the property values and seeing which properties are
dynamic and have an encrypted value. The problem you are highlighting is
that once the component is ghosted, the flow.json/flow.xml now has all
encrypted values and there is no way to get back to the original set of
sensitiveDynamicPropertyNames. For flow.json it would likely be in
VersionedConfigurableExtension where the other property fields are. For
flow.xml, it would require updates to the serialization/deserialization.
When serializing the sensitiveDynamicPropertyNames, it would have to always
retain the set as submitted when the processor was created/updated, and not
base it off the current component's descriptors, otherwise it will have the
same issue.

Curious to see what others think.

Thanks,

Bryan


On Fri, Jun 30, 2023 at 10:46 AM Nissim Shiman <ns...@yahoo.com.invalid>
wrote:

> Hello apache nifi devs,
>
> I have run into an issue that appears to require a different way of
> handling ghost components than we are currently, and would like input
> before proceeding further.
>
>
> Specifically, this is related to
> https://issues.apache.org/jira/browse/NIFI-11570
>
>
> Besides the example in the ticket, a good case where this issue can be
> seen is with the ExecuteScript processor with two dynamic properties, one
> sensitive and one not.  On recovery from ghost component, both properties
> will be sensitive.
>
>
> The challenge is that on recovery from ghost component there no way of
> knowing if a dynamic property is sensitive or not, so it is left in the
> existing sensitive/encrypted state.
>
>
> There are solutions, but each is a shift from how nifi is currently
> working.
>
> 1.  Leave unencrypted properties in the clear when entering ghost
> component mode.  This would work, but we'd lose protections of why
> encryption was done in the first place, so probably a non-starter.
>
> 2.  When encrypting properties in ghost mode, have a custom encryption tag
> in flow.json.gz/flow.xml.gz.   i.e. instead of sensitive values starting
> with enc{   they would start with enc-ghost{   This will help when exiting
> ghost mode where we could now more easily determine why a value was
> encrypted in the first place, but is introducing new complexities in
> keeping track of why something is encrypted
>
>
> 3.  Leave handling of ghost components as is, but for a given component
> only allow Dynamic Properties to be non-sensitive or senstive, but not
> both.  This would affect ExecuteScript, InvokeHTTP (and others) which
> currently allow both and would be rolling back considerable work that was
> done to allow this, and we'd lose existing functionality, so also not
> likely.
>
>
> 4.  Leave as is.  It is not so common to have components in ghost state,
> and only some of them allow Dynamic Properties so this is an acceptable
> risk considering the alternatives.
>
> I am interested in community feedback and/or other suggestions to the
> current situation.
>
>
> Thanks,
>
>
> Nissim Shiman
>
>
>
>
>
>