You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "ValeriiKretinin (via GitHub)" <gi...@apache.org> on 2024/02/29 10:08:44 UTC
[I] List type support for host in connections [airflow]
ValeriiKretinin opened a new issue, #37790:
URL: https://github.com/apache/airflow/issues/37790
### Apache Airflow version
Other Airflow 2 version (please specify below)
### If "Other Airflow 2 version" selected, which one?
2.7.X and 2.8.X
### What happened?
To store secrets and connections we use hashicorp as a secret backend.
After updating airflow from version 2.6.X to 2.7.X and 2.8.X we got the following problem:
```
[2024-02-29, 09:05:06 UTC] {connection.py:471} ERROR - Unable to retrieve connection from secrets backend (VaultBackend). Checking subsequent secrets backend.
Traceback (most recent call last):
File "/home/airflow/.local/lib/python3.11/site-packages/airflow/models/connection.py", line 468, in get_connection_from_secrets
SecretCache.save_connection_uri(conn_id, conn.get_uri())
^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.11/site-packages/airflow/models/connection.py", line 264, in get_uri
host_block += quote(host, safe="")
^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/urllib/parse.py", line 893, in quote
return quote_from_bytes(string, safe)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/urllib/parse.py", line 923, in quote_from_bytes
raise TypeError("quote_from_bytes() expected bytes")
TypeError: quote_from_bytes() expected bytes
[2024-02-29, 09:05:06 UTC] {taskinstance.py:1937} ERROR - Task failed with exception
Traceback (most recent call last):
File "/home/airflow/.local/lib/python3.11/site-packages/airflow/operators/python.py", line 263, in execute
condition = super().execute(context)
^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.11/site-packages/airflow/operators/python.py", line 192, in execute
return_value = self.execute_callable()
^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.11/site-packages/airflow/operators/python.py", line 209, in execute_callable
return self.python_callable(*self.op_args, **self.op_kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/airflow/dags/abnormal_multiples/abnormal_multiples_dag.py", line 110, in generate_abnormal_multiples_to_post_in_slack
client = init_clickhouse_client(clickhouse_connection_id)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/airflow/dags/common/clickhouse_client.py", line 5, in init_clickhouse_client
conn = BaseHook.get_connection(clickhouse_connection)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.11/site-packages/airflow/hooks/base.py", line 72, in get_connection
conn = Connection.get_connection_from_secrets(conn_id)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.11/site-packages/airflow/models/connection.py", line 477, in get_connection_from_secrets
raise AirflowNotFoundException(f"The conn_id `{conn_id}` isn't defined")
airflow.exceptions.AirflowNotFoundException: The conn_id `clickhouse_dwh` isn't defined
```
Based on the analysis of the latest changes here https://github.com/apache/airflow/blob/main/airflow/models/connection.py, it was possible to find out that only the str type was supported, after which list is no longer supported, accordingly, all connections in vault need to be converted to str
If change host from list to a string, the error goes away and everything works correctly again, although this was not required in versions 2.6.X and older
### What you think should happen instead?
It would be great to get back support for type list, since it is more convenient to store several hosts in a list rather than in one line or create different connections to store different hosts
However, if you do not plan to support list or it is technically difficult to implement, the issue can be closed
### How to reproduce
Create a connection in vault and enter a list of hosts in list
```
host: [
host1,
host2....
]
```
### Operating System
k8s helm chart
### Versions of Apache Airflow Providers
apache-airflow-providers-hashicorp 3.6.3
also tried with 3.5.0
### Deployment
Other 3rd-party Helm chart
### Deployment details
community helm chart
### Anything else?
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] List type support for host in connections [airflow]
Posted by "ValeriiKretinin (via GitHub)" <gi...@apache.org>.
ValeriiKretinin closed issue #37790: List type support for host in connections
URL: https://github.com/apache/airflow/issues/37790
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] List type support for host in connections [airflow]
Posted by "ValeriiKretinin (via GitHub)" <gi...@apache.org>.
ValeriiKretinin commented on issue #37790:
URL: https://github.com/apache/airflow/issues/37790#issuecomment-1971044847
Ok, got it, thank you for your answer, I will change my connections to strings, closing this issue
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] List type support for host in connections [airflow]
Posted by "boring-cyborg[bot] (via GitHub)" <gi...@apache.org>.
boring-cyborg[bot] commented on issue #37790:
URL: https://github.com/apache/airflow/issues/37790#issuecomment-1970807785
Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] List type support for host in connections [airflow]
Posted by "Taragolis (via GitHub)" <gi...@apache.org>.
Taragolis commented on issue #37790:
URL: https://github.com/apache/airflow/issues/37790#issuecomment-1970940967
> However, if you do not plan to support list or it is technically difficult to implement, the issue can be closed
I would rather say with current implementation of connections to solve this cases would add more complexity. `airflow.models.connection.Connection` is also DB model and the expectation that the data follow defined schema into the Airflow DB, and into the DB this field is a string/text/varchar and all internal logic of that fields expects to get string here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org