[GitHub] [airflow] Yuvaneshnm opened a new issue #15271: When retrieving connections from AWS secret manger emr_hook.py should convert strings to their expected format

[GitBox <gi...@apache.org>]

Yuvaneshnm opened a new issue #15271:
URL: https://github.com/apache/airflow/issues/15271


   **Description**
   
   When we try to use AWS secret manager as backend to store airflow connections(in URI format), Our DAG fails with error 
   **Error:**
   {{emr_create_job_flow_operator.py:66}} INFO - Creating JobFlow using aws-conn-id: AWS-S3-TCDS, emr-conn-id: AWS-EMR-TCDS-AGG-HOURLY-STG
   [2021-04-02 18:41:19,041] {{taskinstance.py:1150}} ERROR - Parameter validation failed:
   Invalid type for parameter Applications, value: [{'Name': 'Hadoop'}, {'Name': 'Spark'}, {'Name': 'Zeppelin'}, {'Name': 'Hive'}, {'Name': 'Ganglia'}], type: <class 'str'>, valid types: <class 'list'>, <class 'tuple'>
   
   Note : All the values in the connection are retrieved as String and we need emr_hook.py to convert strings to their expected format. 
   
   **Use case / motivation**
   
   We need emr_hook.py to convert strings to their expected format so that we will be able use our connections stored in AWS secret manager backend which is recommended by AWS Managed airflow(MWAA)
   
   
   **Are you willing to submit a PR?**
   
   No
   
   **Related Issues**
   
   Could not find any related issue at this time.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] dstandish closed issue #15271: When retrieving connections from AWS secret manger emr_hook.py should convert strings to their expected format

[GitBox <gi...@apache.org>]

dstandish closed issue #15271:
URL: https://github.com/apache/airflow/issues/15271


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] dstandish commented on issue #15271: When retrieving connections from AWS secret manger emr_hook.py should convert strings to their expected format

[GitBox <gi...@apache.org>]

dstandish commented on issue #15271:
URL: https://github.com/apache/airflow/issues/15271#issuecomment-981874440


   @Yuvaneshnm how did you produce your URI?
   
   Try using `Connection.get_uri` to produce your URI. (first build the connection object in the normal python way e.g. `Connection(password='...', ...)`.
   
   This will produce a URI that can be parsed back to the identical connection.  Update secrets manager to use the URI.
   
   I will close for now but reply if there are issues.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] boring-cyborg[bot] commented on issue #15271: When retrieving connections from AWS secret manger emr_hook.py should convert strings to their expected format

[GitBox <gi...@apache.org>]

boring-cyborg[bot] commented on issue #15271:
URL: https://github.com/apache/airflow/issues/15271#issuecomment-815584936


   Thanks for opening your first issue here! Be sure to follow the issue template!
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org