Fwd: Hashicorp vault transit engine for sensitive properties of processors

[Cannon Palms <cp...@influxdata.com>]

Hello,

From what I understand from the documentation, the transit engine of
Hashicorp Vault is definitely supported for system properties. It is also
clear that the standard key/value engine of Hashicorp vault is supported
for sensitive processor properties (e.g. the password used to connect to an
MQTT broker in a ConsumeMQTT processor).

What I cannot tell is if NiFi supports using the transit engine for these
sensitive properties of processors.

I'd like to ensure that these properties are encrypted at rest inside of
the registry, but decrypted using the transit engine and a provided vault
encryption key at runtime.

Is this currently supported? Or is the only the standard key/value engine
supported for such properties?

Thanks,
Cannon

Re: Hashicorp vault transit engine for sensitive properties of processors

[Joe Gresock <jg...@gmail.com>]

Cannon,

There is also an ongoing effort to add a new framework-level component
called a Parameter Provider [1], which will allow parameter values to be
fetched into a NiFi Parameter Context from an external source.  A HashiCorp
Vault Key/Value Parameter Provider [2] has been proposed and is being
worked.  In this feature, sensitive parameter values are still encrypted as
usual in the actual flow.xml.gz once fetched from the external source, so
it doesn't exactly match your original question, but it seems to be one
step closer.  We hope to include this in a release sometime soon.

[1] https://issues.apache.org/jira/browse/NIFI-8998
[2] https://issues.apache.org/jira/browse/NIFI-9401

Joe

On Thu, Feb 3, 2022 at 9:06 AM David Handermann <ex...@apache.org>
wrote:

> Hi Cannon,
>
> The following NiFi Jira issue outlines a previous attempt to integrate
> flow property storage and retrieval using HashiCorp Vault:
>
> https://issues.apache.org/jira/browse/NIFI-6825
>
> I updated and closed the issue since the description itself outlines an
> approach that no longer fits well in light of recent improvements to the
> framework.
>
> Feel free to create a new issue and we can use that going forward.
>
> Regards,
> David Handermann
>
> On Thu, Feb 3, 2022 at 12:23 AM Cannon Palms <cp...@influxdata.com>
> wrote:
>
>> Thanks Joe! Do you know if there is an existing JIRA issue to track such
>> a feature proposal (vault integration for flow secrets)? I’d be happy to
>> create one if not.
>>
>> Cannon
>>
>> On Tue, Feb 1, 2022 at 3:46 PM Joe Gresock <jg...@gmail.com> wrote:
>>
>>> Hi Cannon,
>>>
>>> Both the HashiCorp Vault Transit and Key/Value Sensitive Property
>>> Providers are able to protect NiFi's configuration files (e.g.,
>>> nifi.properties, login-identity-providers.xml, and authorizers.xml).  In
>>> the case of the Transit implementation, you would use the encrypt-config.sh
>>> tool from the NiFi Toolkit to encrypt properties in these files using the
>>> Vault Transit Engine, and these will be decrypted using Vault as NiFi
>>> starts up.  The process is similar for the Key/Value implementation, but
>>> the values of the properties are stored inside the Vault server instead of
>>> being encrypted at rest in the configuration files.
>>>
>>> The properties in the flow.xml.gz file (e.g., your ConsumeMQTT processor
>>> password) are protected by a different mechanism, and there is not
>>> currently a Vault implementation that protects these.
>>>
>>> Hope this helps,
>>> Joe
>>>
>>> On Tue, Feb 1, 2022 at 11:05 AM Cannon Palms <cp...@influxdata.com>
>>> wrote:
>>>
>>>> Hello,
>>>>
>>>> From what I understand from the documentation, the transit engine of
>>>> Hashicorp Vault is definitely supported for system properties. It is also
>>>> clear that the standard key/value engine of Hashicorp vault is supported
>>>> for sensitive processor properties (e.g. the password used to connect to an
>>>> MQTT broker in a ConsumeMQTT processor).
>>>>
>>>> What I cannot tell is if NiFi supports using the transit engine for
>>>> these sensitive properties of processors.
>>>>
>>>> I'd like to ensure that these properties are encrypted at rest inside
>>>> of the registry, but decrypted using the transit engine and a provided
>>>> vault encryption key at runtime.
>>>>
>>>> Is this currently supported? Or is the only the standard key/value
>>>> engine supported for such properties?
>>>>
>>>> Thanks,
>>>> Cannon
>>>>
>>>

Re: Hashicorp vault transit engine for sensitive properties of processors

[David Handermann <ex...@apache.org>]

Hi Cannon,

The following NiFi Jira issue outlines a previous attempt to integrate flow
property storage and retrieval using HashiCorp Vault:

https://issues.apache.org/jira/browse/NIFI-6825

I updated and closed the issue since the description itself outlines an
approach that no longer fits well in light of recent improvements to the
framework.

Feel free to create a new issue and we can use that going forward.

Regards,
David Handermann

On Thu, Feb 3, 2022 at 12:23 AM Cannon Palms <cp...@influxdata.com> wrote:

> Thanks Joe! Do you know if there is an existing JIRA issue to track such a
> feature proposal (vault integration for flow secrets)? I’d be happy to
> create one if not.
>
> Cannon
>
> On Tue, Feb 1, 2022 at 3:46 PM Joe Gresock <jg...@gmail.com> wrote:
>
>> Hi Cannon,
>>
>> Both the HashiCorp Vault Transit and Key/Value Sensitive Property
>> Providers are able to protect NiFi's configuration files (e.g.,
>> nifi.properties, login-identity-providers.xml, and authorizers.xml).  In
>> the case of the Transit implementation, you would use the encrypt-config.sh
>> tool from the NiFi Toolkit to encrypt properties in these files using the
>> Vault Transit Engine, and these will be decrypted using Vault as NiFi
>> starts up.  The process is similar for the Key/Value implementation, but
>> the values of the properties are stored inside the Vault server instead of
>> being encrypted at rest in the configuration files.
>>
>> The properties in the flow.xml.gz file (e.g., your ConsumeMQTT processor
>> password) are protected by a different mechanism, and there is not
>> currently a Vault implementation that protects these.
>>
>> Hope this helps,
>> Joe
>>
>> On Tue, Feb 1, 2022 at 11:05 AM Cannon Palms <cp...@influxdata.com>
>> wrote:
>>
>>> Hello,
>>>
>>> From what I understand from the documentation, the transit engine of
>>> Hashicorp Vault is definitely supported for system properties. It is also
>>> clear that the standard key/value engine of Hashicorp vault is supported
>>> for sensitive processor properties (e.g. the password used to connect to an
>>> MQTT broker in a ConsumeMQTT processor).
>>>
>>> What I cannot tell is if NiFi supports using the transit engine for
>>> these sensitive properties of processors.
>>>
>>> I'd like to ensure that these properties are encrypted at rest inside of
>>> the registry, but decrypted using the transit engine and a provided vault
>>> encryption key at runtime.
>>>
>>> Is this currently supported? Or is the only the standard key/value
>>> engine supported for such properties?
>>>
>>> Thanks,
>>> Cannon
>>>
>>

Re: Hashicorp vault transit engine for sensitive properties of processors

[Cannon Palms <cp...@influxdata.com>]

Thanks Joe! Do you know if there is an existing JIRA issue to track such a
feature proposal (vault integration for flow secrets)? I’d be happy to
create one if not.

Cannon

On Tue, Feb 1, 2022 at 3:46 PM Joe Gresock <jg...@gmail.com> wrote:

> Hi Cannon,
>
> Both the HashiCorp Vault Transit and Key/Value Sensitive Property
> Providers are able to protect NiFi's configuration files (e.g.,
> nifi.properties, login-identity-providers.xml, and authorizers.xml).  In
> the case of the Transit implementation, you would use the encrypt-config.sh
> tool from the NiFi Toolkit to encrypt properties in these files using the
> Vault Transit Engine, and these will be decrypted using Vault as NiFi
> starts up.  The process is similar for the Key/Value implementation, but
> the values of the properties are stored inside the Vault server instead of
> being encrypted at rest in the configuration files.
>
> The properties in the flow.xml.gz file (e.g., your ConsumeMQTT processor
> password) are protected by a different mechanism, and there is not
> currently a Vault implementation that protects these.
>
> Hope this helps,
> Joe
>
> On Tue, Feb 1, 2022 at 11:05 AM Cannon Palms <cp...@influxdata.com>
> wrote:
>
>> Hello,
>>
>> From what I understand from the documentation, the transit engine of
>> Hashicorp Vault is definitely supported for system properties. It is also
>> clear that the standard key/value engine of Hashicorp vault is supported
>> for sensitive processor properties (e.g. the password used to connect to an
>> MQTT broker in a ConsumeMQTT processor).
>>
>> What I cannot tell is if NiFi supports using the transit engine for these
>> sensitive properties of processors.
>>
>> I'd like to ensure that these properties are encrypted at rest inside of
>> the registry, but decrypted using the transit engine and a provided vault
>> encryption key at runtime.
>>
>> Is this currently supported? Or is the only the standard key/value engine
>> supported for such properties?
>>
>> Thanks,
>> Cannon
>>
>

Re: Hashicorp vault transit engine for sensitive properties of processors

[Joe Gresock <jg...@gmail.com>]

Hi Cannon,

Both the HashiCorp Vault Transit and Key/Value Sensitive Property Providers
are able to protect NiFi's configuration files (e.g., nifi.properties,
login-identity-providers.xml, and authorizers.xml).  In the case of the
Transit implementation, you would use the encrypt-config.sh tool from the
NiFi Toolkit to encrypt properties in these files using the Vault Transit
Engine, and these will be decrypted using Vault as NiFi starts up.  The
process is similar for the Key/Value implementation, but the values of the
properties are stored inside the Vault server instead of being encrypted at
rest in the configuration files.

The properties in the flow.xml.gz file (e.g., your ConsumeMQTT processor
password) are protected by a different mechanism, and there is not
currently a Vault implementation that protects these.

Hope this helps,
Joe

On Tue, Feb 1, 2022 at 11:05 AM Cannon Palms <cp...@influxdata.com> wrote:

> Hello,
>
> From what I understand from the documentation, the transit engine of
> Hashicorp Vault is definitely supported for system properties. It is also
> clear that the standard key/value engine of Hashicorp vault is supported
> for sensitive processor properties (e.g. the password used to connect to an
> MQTT broker in a ConsumeMQTT processor).
>
> What I cannot tell is if NiFi supports using the transit engine for these
> sensitive properties of processors.
>
> I'd like to ensure that these properties are encrypted at rest inside of
> the registry, but decrypted using the transit engine and a provided vault
> encryption key at runtime.
>
> Is this currently supported? Or is the only the standard key/value engine
> supported for such properties?
>
> Thanks,
> Cannon
>