Nifi toolkit encrypt single properties

[Isha Lamboo <is...@virtualsciences.nl>]

Hi all,

I hope this question is appropriate for the developers list, if not, I’ll move it to users.

I have an Ansible role for NiFi that includes generating the NiFi properties files from templates and variables set per cluster/host as well as updating keystore etc.
This works very well, except when it comes to protected properties as set by the toolkit. The toolkit wants to operate on the actual properties files, which causes Ansible to then see differences and want to reset.
I tried with an intermediate processing dir, but then I lose the ability to use Ansible’s –check and –diff options to see if any changes were made. In the end, I added the encrypted values by hand to the Ansible variables files.

As a next step, I’m looking at whether I can use the toolkit’s java classes in my own wrapper to allow me to pass in a master key, protection scheme and raw value and get out the encrypted one, so that I can easily update my encrypted values in the Ansible inventory.
However, my Java skills are somewhat limited, so I would like to ask first if this is even a good idea.

Is this a sensible idea or does it conflict with NiFi or encryption design principles I’m not aware of?

Kind regards,

Isha Lamboo

RE: Nifi toolkit encrypt single properties

[Isha Lamboo <is...@virtualsciences.nl>]

Hi David,

Thanks for your answer.  

I had considered Hashicorp Vault for exactly the reason you state, but at this point it's too much overhead and not all of our NiFi clusters are at 1.14+ yet. You make a good point about the likelihood of changes "under the hood" causing problems. I'll focus instead on invoking encrypt-config.sh in the usual way and extracting the results from the output file somehow. Perhaps I can generate an input with a single property in it and get a single output.

Kind regards,

Isha

-----Oorspronkelijk bericht-----
Van: David Handermann <ex...@apache.org> 
Verzonden: dinsdag 25 januari 2022 15:25
Aan: dev@nifi.apache.org
Onderwerp: Re: Nifi toolkit encrypt single properties

Hi Isha,

Thanks for contacting the developers list and describing what you are attempting to accomplish.

Although there are several Java interfaces and implementation classes that could be leveraged to implement a custom approach to property encryption, that may not be the most reliable method.  NiFi does not consider the Sensitive Property Provider interfaces and classes part of the public API, which means they could be subject to change in minor version updates.  That being said, the input and output format for encrypted property values will remain supported across versions, so you could use the NiFi classes as a starting point. NiFi stores the serialized encrypted values in a custom format containing the initialization vector, followed by a delimiter, and then the encrypted content. It is possible to use standard AES-GCM support in other languages to read the value if you have the root key available.

Recent versions of NiFi introduced support for additional nifi.properties protection schemes using external services such as HashiCorp Vault and Amazon Secrets Manager, as well as other service providers.  The effective result is similar in that nifi.properties contains encrypted or placeholder values, but that might provide some different options.  For example, the HashiCorp Vault Key Value store implementation uses a placeholder value for the property, instead of the encrypted string.  This approach yields the same placeholder value, since HashiCorp Vault stores the actual property.
That kind of strategy requires additional infrastructure, so that may not be a good fit for your environment, but it might be worth considering.

The most stable integration from the perspective of an external tool is the encrypt-config.sh command and associated arguments.

Regards,
David Handermann

On Tue, Jan 25, 2022 at 2:42 AM Isha Lamboo <is...@virtualsciences.nl>
wrote:

> Hi all,
>
> I hope this question is appropriate for the developers list, if not, 
> I’ll move it to users.
>
> I have an Ansible role for NiFi that includes generating the NiFi 
> properties files from templates and variables set per cluster/host as 
> well as updating keystore etc.
> This works very well, except when it comes to protected properties as 
> set by the toolkit. The toolkit wants to operate on the actual 
> properties files, which causes Ansible to then see differences and want to reset.
> I tried with an intermediate processing dir, but then I lose the 
> ability to use Ansible’s –check and –diff options to see if any changes were made.
> In the end, I added the encrypted values by hand to the Ansible 
> variables files.
>
> As a next step, I’m looking at whether I can use the toolkit’s java 
> classes in my own wrapper to allow me to pass in a master key, 
> protection scheme and raw value and get out the encrypted one, so that 
> I can easily update my encrypted values in the Ansible inventory.
> However, my Java skills are somewhat limited, so I would like to ask 
> first if this is even a good idea.
>
> Is this a sensible idea or does it conflict with NiFi or encryption 
> design principles I’m not aware of?
>
> Kind regards,
>
> Isha Lamboo
>
>

Re: Nifi toolkit encrypt single properties

[David Handermann <ex...@apache.org>]

Hi Isha,

Thanks for contacting the developers list and describing what you are
attempting to accomplish.

Although there are several Java interfaces and implementation classes that
could be leveraged to implement a custom approach to property encryption,
that may not be the most reliable method.  NiFi does not consider the
Sensitive Property Provider interfaces and classes part of the public API,
which means they could be subject to change in minor version updates.  That
being said, the input and output format for encrypted property values will
remain supported across versions, so you could use the NiFi classes as a
starting point. NiFi stores the serialized encrypted values in a custom
format containing the initialization vector, followed by a delimiter, and
then the encrypted content. It is possible to use standard AES-GCM support
in other languages to read the value if you have the root key available.

Recent versions of NiFi introduced support for additional nifi.properties
protection schemes using external services such as HashiCorp Vault and
Amazon Secrets Manager, as well as other service providers.  The effective
result is similar in that nifi.properties contains encrypted or placeholder
values, but that might provide some different options.  For example, the
HashiCorp Vault Key Value store implementation uses a placeholder value for
the property, instead of the encrypted string.  This approach yields the
same placeholder value, since HashiCorp Vault stores the actual property.
That kind of strategy requires additional infrastructure, so that may not
be a good fit for your environment, but it might be worth considering.

The most stable integration from the perspective of an external tool is the
encrypt-config.sh command and associated arguments.

Regards,
David Handermann

On Tue, Jan 25, 2022 at 2:42 AM Isha Lamboo <is...@virtualsciences.nl>
wrote:

> Hi all,
>
> I hope this question is appropriate for the developers list, if not, I’ll
> move it to users.
>
> I have an Ansible role for NiFi that includes generating the NiFi
> properties files from templates and variables set per cluster/host as well
> as updating keystore etc.
> This works very well, except when it comes to protected properties as set
> by the toolkit. The toolkit wants to operate on the actual properties
> files, which causes Ansible to then see differences and want to reset.
> I tried with an intermediate processing dir, but then I lose the ability
> to use Ansible’s –check and –diff options to see if any changes were made.
> In the end, I added the encrypted values by hand to the Ansible variables
> files.
>
> As a next step, I’m looking at whether I can use the toolkit’s java
> classes in my own wrapper to allow me to pass in a master key, protection
> scheme and raw value and get out the encrypted one, so that I can easily
> update my encrypted values in the Ansible inventory.
> However, my Java skills are somewhat limited, so I would like to ask first
> if this is even a good idea.
>
> Is this a sensible idea or does it conflict with NiFi or encryption design
> principles I’m not aware of?
>
> Kind regards,
>
> Isha Lamboo
>
>