[GitHub] [airflow] john-jac commented on pull request #19324: Catch AccessDeniedException in AWS Secrets Manager Backend

[GitBox <gi...@apache.org>]

john-jac commented on pull request #19324:
URL: https://github.com/apache/airflow/pull/19324#issuecomment-956325118


   > > Today, this causes a failure with the Scheduler as an uncaught exception
   > 
   > What do you mean with the scheduler? Perhaps most commonly one retrieves a connection from a celery worker. Are you running local executor? Or are you trying to use secrets backend for the metastore conn uri? I.e. was this a task that failed? Or some other scenario?
   
   When you specify a secrets backend, all requests for variables, configs, or connections go through that backend first.  There are numerous errors with an uncaught exception from the secrets backend, including Scheduler, Worker, and Web Server
   
   > Can you help me understand what's the use case where you want to have the retrieval silently fail if access is denied? In this scenario do you store the secret in env vars?
   > 
   > In the "not found" scenario, the reason we suppress is secrets backend goes to the next secrets source (e.g. env var or metastore) when cred not found in the alt backend.
   
   The boto3 secrets manager library will fail with ResourceNotFoundException if there are no restrictions in the IAM role, i.e. it has full access to secrets manager.  That isn't practical for most organizations, as they will have several entities accessing secrets manager and do not want to give full access to all secrets by all of them.  As such, organizations use IAM restrictions tied to a role ARN (i.e. role_arn in AWS connection type) to ensure that each entity only accesses what is allowed.  For example, I may restrict an Airflow environment to only see secrets with the defined connections_prefix:,variables_prefix, and config_prefix.  However, when you do so you do not get a ResourceNotFoundException on missing secrets, which as you indicate should go to the next secrets source, but rather you get AccessDeniedException, which is an uncaught exception and does not go to the next secrets source and fails to retrieve the desired value.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org