How can NIFI dynamically update SAS Tokens for Azure Blob Storage processors.

[Mike Rutlin <md...@gmail.com>]

I have a general question on how to dynamically update and sync SAS tokens
being utilized by the AzureBlogStorage processors ( List,Fetch,Delete, etc)
.

From our NIFI we are accessing a storage container in a different cloud and
thus cannot use AZURE MI as standard for storage containers in the same
cloud. Thus we need to use the SAS Tokens attribute / parameter supported
by the controller service or by the processors.

We pull the SAS token on a periodic basis, before the old SAS token
expires.

In manual testing of this we set a sensitive Parameter to the SAS token and
set up the Controller service SAS Token attribute / parameter to use the
Parameter we created.  Everything worked as it should.

The problem/issue occurs when the SAS token expires and we pull the new
one. How do I modify the Parameter ? Using the NIFI API seems messy and
potential to corrupt the flow configuration. along with the fact that it
stops and starts service / processors with the Parameter we are updating.

I would like to use a flowfile attribute ( which the documentation suggests
is possible ). However the ListAzureBlobStorage processor is a start flow
processor ( no inputs ). Thus I can't run a script to get the new key and
set an attribute that could then be used by the SAS Token attribute /
parameter in the processor.

Any ideas on how I can do this? It seems this would be a common practice to
roll keys and then inject the new key into the data flow for either AWS or
Azure.

If the ListAzureBlobStoreage had an input I coud update with new keys OR if
the list processor had a failure output I could update the key upon
expiration.

Thanks for any and all guidance.

Mike R

Re: How can NIFI dynamically update SAS Tokens for Azure Blob Storage processors.

[Kevin Doran <kd...@apache.org>]

I agree, the Controller Service needs to be enhanced to support rotating
SAS tokens. I know of a few users that have run into this exact problem.

Technically, it should possible to write a custom implementation of
the ADLSCredentialsService API that did this, but it would be ideal if NiFi
provided some capability out of the box.

On Mar 13, 2023 at 21:28:33, Adam Taft <ad...@adamtaft.com> wrote:

> Hi Mike,
>
> You're right, I don't think you're going to get anywhere with the
> flowfile attribute providing the value, as like you said,
> ListAzureBlobStorage is a source processor and doesn't accept any input
> flowfiles.
>
> So currently, you're really looking at two choices, which you've already
> identified:
> 1.  Manually rotate the SAS token through the user interface.
> 2.  Make an HTTP call to the NiFI API causing an update to the controller
> service SAS token property.
>
> Now, #2 is not quite as bad as you think it is. Luckily, NiFi will deal
> very nicely with the temporary restart of the backing controller service.
> NiFi handles shutting down all the processors that are connected to the
> controller service and then bringing them back up (from multiple API
> calls). It will queue up the flowfiles for the short time this operation
> occurs (but yes, it does need to stop those processors before any changes
> can be made). In theory, you shouldn't have any corruption issues going
> this path. The problem side of it will be in writing your script to connect
> with the API and issuing the request to make this change. A bit painful,
> but possible. Your script would need to locate or know how to find the
> correct controller service (typically via its uuid).
>
> All this really speaks to a feature request, to be honest. It would be
> ideal that the controller service can reconfigure itself by grabbing the
> new SAS token for your storage account from Azure. Presumably the best
> practice would be to pick up the new SAS token from Key Vault (since Key
> Vault can be configured to own SAS Token generation for the storage
> account)? I guess Azure AD / OAuth is also a possible (alternative)
> solution?
>
> Anyway, short term, consider the API approach. Long term, a contribution
> to the project along these lines would probably be gratefully accepted. At
> minimum, a Jira ticket to suggest the new feature would be a good starting
> place.
>
> /Adam
>
>
>
> On Wed, Mar 8, 2023 at 9:53 AM Mike Rutlin <md...@gmail.com> wrote:
>
>>
>> I have a general question on how to dynamically update and sync SAS
>> tokens being utilized by the AzureBlogStorage processors (
>> List,Fetch,Delete, etc) .
>>
>> From our NIFI we are accessing a storage container in a different cloud
>> and thus cannot use AZURE MI as standard for storage containers in the same
>> cloud. Thus we need to use the SAS Tokens attribute / parameter supported
>> by the controller service or by the processors.
>>
>> We pull the SAS token on a periodic basis, before the old SAS token
>> expires.
>>
>> In manual testing of this we set a sensitive Parameter to the SAS token
>> and set up the Controller service SAS Token attribute / parameter to use
>> the Parameter we created.  Everything worked as it should.
>>
>> The problem/issue occurs when the SAS token expires and we pull the new
>> one. How do I modify the Parameter ? Using the NIFI API seems messy and
>> potential to corrupt the flow configuration. along with the fact that it
>> stops and starts service / processors with the Parameter we are updating.
>>
>> I would like to use a flowfile attribute ( which the documentation
>> suggests is possible ). However the ListAzureBlobStorage processor is a
>> start flow processor ( no inputs ). Thus I can't run a script to get the
>> new key and set an attribute that could then be used by the SAS Token
>> attribute / parameter in the processor.
>>
>> Any ideas on how I can do this? It seems this would be a common practice
>> to roll keys and then inject the new key into the data flow for either AWS
>> or Azure.
>>
>> If the ListAzureBlobStoreage had an input I coud update with new keys OR
>> if the list processor had a failure output I could update the key upon
>> expiration.
>>
>> Thanks for any and all guidance.
>>
>> Mike R
>>
>>

Re: How can NIFI dynamically update SAS Tokens for Azure Blob Storage processors.

[Adam Taft <ad...@adamtaft.com>]

Hi Mike,

You're right, I don't think you're going to get anywhere with the flowfile
attribute providing the value, as like you said, ListAzureBlobStorage is a
source processor and doesn't accept any input flowfiles.

So currently, you're really looking at two choices, which you've already
identified:
1.  Manually rotate the SAS token through the user interface.
2.  Make an HTTP call to the NiFI API causing an update to the controller
service SAS token property.

Now, #2 is not quite as bad as you think it is. Luckily, NiFi will deal
very nicely with the temporary restart of the backing controller service.
NiFi handles shutting down all the processors that are connected to the
controller service and then bringing them back up (from multiple API
calls). It will queue up the flowfiles for the short time this operation
occurs (but yes, it does need to stop those processors before any changes
can be made). In theory, you shouldn't have any corruption issues going
this path. The problem side of it will be in writing your script to connect
with the API and issuing the request to make this change. A bit painful,
but possible. Your script would need to locate or know how to find the
correct controller service (typically via its uuid).

All this really speaks to a feature request, to be honest. It would be
ideal that the controller service can reconfigure itself by grabbing the
new SAS token for your storage account from Azure. Presumably the best
practice would be to pick up the new SAS token from Key Vault (since Key
Vault can be configured to own SAS Token generation for the storage
account)? I guess Azure AD / OAuth is also a possible (alternative)
solution?

Anyway, short term, consider the API approach. Long term, a contribution to
the project along these lines would probably be gratefully accepted. At
minimum, a Jira ticket to suggest the new feature would be a good starting
place.

/Adam



On Wed, Mar 8, 2023 at 9:53 AM Mike Rutlin <md...@gmail.com> wrote:

>
> I have a general question on how to dynamically update and sync SAS tokens
> being utilized by the AzureBlogStorage processors ( List,Fetch,Delete, etc)
> .
>
> From our NIFI we are accessing a storage container in a different cloud
> and thus cannot use AZURE MI as standard for storage containers in the same
> cloud. Thus we need to use the SAS Tokens attribute / parameter supported
> by the controller service or by the processors.
>
> We pull the SAS token on a periodic basis, before the old SAS token
> expires.
>
> In manual testing of this we set a sensitive Parameter to the SAS token
> and set up the Controller service SAS Token attribute / parameter to use
> the Parameter we created.  Everything worked as it should.
>
> The problem/issue occurs when the SAS token expires and we pull the new
> one. How do I modify the Parameter ? Using the NIFI API seems messy and
> potential to corrupt the flow configuration. along with the fact that it
> stops and starts service / processors with the Parameter we are updating.
>
> I would like to use a flowfile attribute ( which the documentation
> suggests is possible ). However the ListAzureBlobStorage processor is a
> start flow processor ( no inputs ). Thus I can't run a script to get the
> new key and set an attribute that could then be used by the SAS Token
> attribute / parameter in the processor.
>
> Any ideas on how I can do this? It seems this would be a common practice
> to roll keys and then inject the new key into the data flow for either AWS
> or Azure.
>
> If the ListAzureBlobStoreage had an input I coud update with new keys OR
> if the list processor had a failure output I could update the key upon
> expiration.
>
> Thanks for any and all guidance.
>
> Mike R
>
>