You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "dviru (via GitHub)" <gi...@apache.org> on 2023/05/16 07:31:25 UTC
[GitHub] [airflow] dviru opened a new issue, #31310: Issue while getting connection using basehook
dviru opened a new issue, #31310:
URL: https://github.com/apache/airflow/issues/31310
### Apache Airflow version
Other Airflow 2 version (please specify below)
### What happened
This issue found in airflow 2.4.3 version
Currently we are using airflow 2.2.5 and planning to upgrade to 2.4.3, While testing our dags and code we found issue in airflow 2.4.3. In one of our usage cases we are getting the connection details using below code(We are using SECRETS__BACKEND as airflow.providers.amazon.aws.secrets.systems_manager.SystemsManagerParameterStoreBackend)
`api_conn = BaseHook.get_connection('api_yield_analytics')`
But above code is behaving differently in 2.2.5 and 2.4.3. In airflow 2.4.3, it is removing the first character from the connection for example if my connection is "**username:password?schema=airflow**", in airflow 2.4.3 it is giving "**sername:password?schema=airflow**" but in airflow 2.2.5 it is returning the properly.
When I debug something more, I found the below observations.
1. In BaseSecretsBackend class, fetching connection value with extra space at beginning of the the connection value. for example " **username:password?schema=airflow**" in get_connection method(Line number 106 in 2.4.3 version and line number 64 in 2.2.5 version)
2. In airflow 2.4.3 we are using strip method on the connection value(base_secrets.py line-number 65), but not in airflow 2.2.5. So our connection value will be as given below
`
In 2.4.3 "username:password?schema=airflow"(Space removed from the beginning of the connection)
in 2.2.5 " username:password?schema=airflow"(Space not removed from the beginning of the connection)
`
3. Create the Connection object from the connection URL .
4. In Connection in class we have one method called "**_parse_from_uri**", in this method we have one statement "**quoted_schema = uri_parts.path[1:]**", this line of code removing the first character from the code. So in airflow 2.2.5 it removes the SPACE from the connection value but in airflow 2.4.3 remove real character from the connection.
Because of this behaviour I am facing issue in one of our DAG.
Please let me know if my understanding is wrong or any other suggestion how to avoid this issue.
### What you think should happen instead
If my connection value is "**username:password?schema=airflow**" then it should return "**username:password?schema=airflow**" instead of " **username:password?schema=airflow**"
### How to reproduce
Store the connection in AWS parameter store and execute the below code
from airflow.hooks.base import BaseHook
api_conn = BaseHook.get_connection('api_conn')
print(api_conn.schema)
### Operating System
Linux
### Versions of Apache Airflow Providers
_No response_
### Deployment
Other Docker-based deployment
### Deployment details
_No response_
### Anything else
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #31310: Issue while getting connection using basehook
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on issue #31310:
URL: https://github.com/apache/airflow/issues/31310#issuecomment-1549203930
The special character might include space BTW.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #31310: Issue while getting connection using basehook
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on issue #31310:
URL: https://github.com/apache/airflow/issues/31310#issuecomment-1549202094
You need to percent-endode components of your URL, otherwise it is not a valid URL. see https://datatracker.ietf.org/doc/html/rfc3986#section-2.4 and also this chapter that explains how to build percent-encoded URI https://airflow.apache.org/docs/apache-airflow/stable/howto/connection.html#uri-format
I think there is something wrong with your URI (contains `%` or other special character in the password or similar)
It could have worked in your special case correctly in 2.2.5 but that was likely accidental and you need to fixe it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk closed issue #31310: Issue while getting connection using basehook
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk closed issue #31310: Issue while getting connection using basehook
URL: https://github.com/apache/airflow/issues/31310
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] dviru commented on issue #31310: Issue while getting connection using basehook
Posted by "dviru (via GitHub)" <gi...@apache.org>.
dviru commented on issue #31310:
URL: https://github.com/apache/airflow/issues/31310#issuecomment-1549258667
Sure. Thanks @potiuk
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org