You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "joao-miranda (via GitHub)" <gi...@apache.org> on 2023/05/24 08:02:22 UTC

[GitHub] [airflow] joao-miranda opened a new issue, #31496: Can't retrieve variable stored in AWS SecretsManager in DAG in MWAA (Airflow 2.5.1)

joao-miranda opened a new issue, #31496:
URL: https://github.com/apache/airflow/issues/31496

   ### Apache Airflow version
   
   Other Airflow 2 version (please specify below)
   
   ### What happened
   
   Airflow 2.5.1
   
   I've followed the instructions I found online, for example https://blog.beachgeek.co.uk/working-with-parameters-and-variables-in-amazon-managed-workflows-for-apache-airflow/
   I'm storing a variable encrypted by KMS in SecretsManager with name **airflow/variables/AAA.**
   
    
   This configuration is present in the Airflow Environment:
   ```
   secrets.backend | airflow.providers.amazon.aws.secrets.secrets_manager.SecretsManagerBackend
   secrets.backend_kwargs | { "connections_prefix": "airflow/connections", "variables_prefix": "airflow/variables" }
   ```
   
   Airflow IAM user has full KMS and SecretsManager permissions.
   
   Trying to get the value from the code like this is not working:
   ```
   AAA = Variable.get('AAA')
   ```
   
   The DAG import fails with this error:
   ```
   Broken DAG: [/usr/local/airflow/dags/pipeline.py] Traceback (most recent call last):
     File "/usr/local/airflow/dags/pipeline.py", line 14, in <module>
       AAA = Variable.get('AAA')
     File "/usr/local/airflow/.local/lib/python3.10/site-packages/airflow/models/variable.py", line 141, in get
       raise KeyError(f"Variable {key} does not exist")
   KeyError: 'Variable AAA does not exist'
   ```
   
   ### What you think should happen instead
   
   Value should be retrieved from Secrets Manager
   
   ### How to reproduce
   
   The above code in MWAA should replicate the same behaviour.
   
   ### Operating System
   
   AWS
   
   ### Versions of Apache Airflow Providers
   
   AWS MWAA (Apache Airflow 2.5.1)
   
   ### Deployment
   
   Amazon (AWS) MWAA
   
   ### Deployment details
   
   _No response_
   
   ### Anything else
   
   _No response_
   
   ### Are you willing to submit PR?
   
   - [X] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] eladkal commented on issue #31496: Can't retrieve variable stored in AWS SecretsManager in DAG in MWAA (Airflow 2.5.1)

Posted by "eladkal (via GitHub)" <gi...@apache.org>.
eladkal commented on issue #31496:
URL: https://github.com/apache/airflow/issues/31496#issuecomment-1561036131

   The report doesn't clarify what is the issue.
   What exactly is the bug?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] eladkal closed issue #31496: Can't retrieve variable stored in AWS SecretsManager in DAG in MWAA (Airflow 2.5.1)

Posted by "eladkal (via GitHub)" <gi...@apache.org>.
eladkal closed issue #31496: Can't retrieve variable stored in AWS SecretsManager in DAG in MWAA (Airflow 2.5.1)
URL: https://github.com/apache/airflow/issues/31496


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] boring-cyborg[bot] commented on issue #31496: Can't retrieve variable stored in AWS SecretsManager in DAG in MWAA (Airflow 2.5.1)

Posted by "boring-cyborg[bot] (via GitHub)" <gi...@apache.org>.
boring-cyborg[bot] commented on issue #31496:
URL: https://github.com/apache/airflow/issues/31496#issuecomment-1560637315

   Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] vincbeck commented on issue #31496: Can't retrieve variable stored in AWS SecretsManager in DAG in MWAA (Airflow 2.5.1)

Posted by "vincbeck (via GitHub)" <gi...@apache.org>.
vincbeck commented on issue #31496:
URL: https://github.com/apache/airflow/issues/31496#issuecomment-1561833164

   Are you sure the AWS connection points to the right account and region? The message `Variable AAA does not exist` makes me thing the AWS connection targets and account/region where the variable is not defined


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] eladkal commented on issue #31496: Can't retrieve variable stored in AWS SecretsManager in DAG in MWAA (Airflow 2.5.1)

Posted by "eladkal (via GitHub)" <gi...@apache.org>.
eladkal commented on issue #31496:
URL: https://github.com/apache/airflow/issues/31496#issuecomment-1561259016

   Please add reproduce example


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] joao-miranda commented on issue #31496: Can't retrieve variable stored in AWS SecretsManager in DAG in MWAA (Airflow 2.5.1)

Posted by "joao-miranda (via GitHub)" <gi...@apache.org>.
joao-miranda commented on issue #31496:
URL: https://github.com/apache/airflow/issues/31496#issuecomment-1561142599

   The bug is that integration between Airflow and AWS SecretsManager seems to be broken. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org