You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "agreenburg (via GitHub)" <gi...@apache.org> on 2024/09/24 18:54:53 UTC
[I] MySqlHook constructor fails with exception after upgrade from apache-airflow-providers-common-sql from 1.16.0 to 1.17.0 [airflow]
agreenburg opened a new issue, #42452:
URL: https://github.com/apache/airflow/issues/42452
### Apache Airflow Provider(s)
common-sql, mysql
### Versions of Apache Airflow Providers
apache-airflow-providers-common-sql 1.17.0
apache-airflow-providers-mysql 5.6.1
### Apache Airflow version
2.9.2+astro.1
### Operating System
debian astronomer/astro-runtime:11.5.0
### Deployment
Astronomer
### Deployment details
This is an Astronomer-hosted deployment. Docker image is quay.io/astronomer/astro-runtime:11.5.0
### What happened
DAGs that interact with MySqlHook started failing after apache-airflow-providers-common-sql was automatically upgraded from 1.16.0 to 1.17.0.
Here is an example error from the log:
```
[2024-09-24, 18:24:30 UTC] {taskinstance.py:2905} ERROR - Task failed with exception
Traceback (most recent call last):
File "/usr/local/lib/python3.11/site-packages/airflow/models/taskinstance.py", line 465, in _execute_task
result = _execute_callable(context=context, **execute_callable_kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/airflow/models/taskinstance.py", line 432, in _execute_callable
return execute_callable(context=context, **execute_callable_kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/airflow/models/baseoperator.py", line 401, in wrapper
return func(self, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/airflow/operators/python.py", line 235, in execute
return_value = self.execute_callable()
^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/airflow/operators/python.py", line 252, in execute_callable
return self.python_callable(*self.op_args, **self.op_kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/airflow/dags/sftp_handler.py", line 25, in getDBConnection
return MySqlHook(connection_name).get_conn()
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/airflow/providers/mysql/hooks/mysql.py", line 75, in __init__
self.connection = kwargs.pop("connection", None)
^^^^^^^^^^^^^^^
AttributeError: property 'connection' of 'MySqlHook' object has no setter
```
requirements.txt had apache-airflow-providers-mysql with no version constraint.
```
apache-airflow-providers-mysql
```
Forcing downgrade to the previous version fixes the problem and allows the DAGs to run again:
```
apache-airflow-providers-mysql
apache-airflow-providers-common-sql <1.17
```
### What you think should happen instead
Because we are including apache-airflow-providers-mysql without a version constraint, I would hope and expect that it would only bring in a version of apache-airflow-providers-common-sql that is compatible with the version of apache-airflow-providers-mysql that is being used. That does not appear to be happening here.
### How to reproduce
1. Install these exact dependency versions:
apache-airflow-providers-common-sql 1.17.0
apache-airflow-providers-mysql 5.6.1
2. Inside a python operator, attempt to get an instance of a MysqlHook with a named connection:
```
conn = MySqlHook("my_named_mysql_connection").get_conn()
```
This will throw an AttributeError:
```
AttributeError: property 'connection' of 'MySqlHook' object has no setter
```
### Anything else
This is consistently reproducible with the listed provider versions.
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] MySqlHook constructor fails with exception after upgrading apache-airflow-providers-common-sql from 1.16.0 to 1.17.0 [airflow]
Posted by "dabla (via GitHub)" <gi...@apache.org>.
dabla commented on issue #42452:
URL: https://github.com/apache/airflow/issues/42452#issuecomment-2376294980
This is probably due to my [PR 40751](https://github.com/apache/airflow/pull/40751), a dummy setter would indeed solve the issue
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] MySqlHook constructor fails with exception after upgrade from apache-airflow-providers-common-sql from 1.16.0 to 1.17.0 [airflow]
Posted by "boring-cyborg[bot] (via GitHub)" <gi...@apache.org>.
boring-cyborg[bot] commented on issue #42452:
URL: https://github.com/apache/airflow/issues/42452#issuecomment-2372065313
Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] MySqlHook constructor fails with exception after upgrading apache-airflow-providers-common-sql from 1.16.0 to 1.17.0 [airflow]
Posted by "Lee-W (via GitHub)" <gi...@apache.org>.
Lee-W commented on issue #42452:
URL: https://github.com/apache/airflow/issues/42452#issuecomment-2375809304
I'll send a PR for adding a dummy setter for backcompat.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] MySqlHook constructor fails with exception after upgrading apache-airflow-providers-common-sql from 1.16.0 to 1.17.0 [airflow]
Posted by "tboschtxr (via GitHub)" <gi...@apache.org>.
tboschtxr commented on issue #42452:
URL: https://github.com/apache/airflow/issues/42452#issuecomment-2375047098
@rawwar this did work yes, thank you!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] MySqlHook constructor fails with exception after upgrading apache-airflow-providers-common-sql from 1.16.0 to 1.17.0 [airflow]
Posted by "dweaver33 (via GitHub)" <gi...@apache.org>.
dweaver33 commented on issue #42452:
URL: https://github.com/apache/airflow/issues/42452#issuecomment-2373860412
+1, we're having the same issue as well
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] MySqlHook constructor fails with exception after upgrading apache-airflow-providers-common-sql from 1.16.0 to 1.17.0 [airflow]
Posted by "tboschtxr (via GitHub)" <gi...@apache.org>.
tboschtxr commented on issue #42452:
URL: https://github.com/apache/airflow/issues/42452#issuecomment-2374924095
+1 same issue but for postgres
``` File "/usr/local/lib/python3.10/site-packages/airflow/providers/postgres/hooks/postgres.py", line 90, in __init__
self.connection: Connection | None = kwargs.pop("connection", None)
AttributeError: can't set attribute 'connection'```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] MySqlHook constructor fails with exception after upgrading apache-airflow-providers-common-sql from 1.16.0 to 1.17.0 [airflow]
Posted by "veer0318 (via GitHub)" <gi...@apache.org>.
veer0318 commented on issue #42452:
URL: https://github.com/apache/airflow/issues/42452#issuecomment-2373995902
+1 same issue
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] MySqlHook constructor fails with exception after upgrading apache-airflow-providers-common-sql from 1.16.0 to 1.17.0 [airflow]
Posted by "tboschtxr (via GitHub)" <gi...@apache.org>.
tboschtxr commented on issue #42452:
URL: https://github.com/apache/airflow/issues/42452#issuecomment-2374980584
@rawwar
I will give this a try now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] MySqlHook constructor fails with exception after upgrading apache-airflow-providers-common-sql from 1.16.0 to 1.17.0 [airflow]
Posted by "phanikumv (via GitHub)" <gi...@apache.org>.
phanikumv closed issue #42452: MySqlHook constructor fails with exception after upgrading apache-airflow-providers-common-sql from 1.16.0 to 1.17.0
URL: https://github.com/apache/airflow/issues/42452
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] MySqlHook constructor fails with exception after upgrading apache-airflow-providers-common-sql from 1.16.0 to 1.17.0 [airflow]
Posted by "Lee-W (via GitHub)" <gi...@apache.org>.
Lee-W commented on issue #42452:
URL: https://github.com/apache/airflow/issues/42452#issuecomment-2375782275
Just traced the code a bit. This was due to a change in `apache-airflow-providers-mysql==1.17.0`. [connection](https://github.com/apache/airflow/pull/40751/files#diff-6e1b2f961cb951d05d66d2d814ef5f6d8f8bf8f43c40fb5d40e27a031fed8dd7R208) is now a descriptor without a setter (and I think this is by design).
The reason why `apache-airflow-providers-mysql==5.7.1` works is due to the fix in the same [PR](https://github.com/apache/airflow/pull/40751/files#diff-b072809da92f96948d535afe96b8b6b1927a5ecca99144eaf79a603449a29406L75). Also postgres change [here](https://github.com/apache/airflow/pull/40751/files#diff-f1ec9506a1f434e38d095c1f10634243f4cce602a595cd786f0ca33aededb051L91) along with other providers in that PR
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] MySqlHook constructor fails with exception after upgrading apache-airflow-providers-common-sql from 1.16.0 to 1.17.0 [airflow]
Posted by "agreenburg (via GitHub)" <gi...@apache.org>.
agreenburg commented on issue #42452:
URL: https://github.com/apache/airflow/issues/42452#issuecomment-2375076114
This looks like it is resolved with the latest version of apache-airflow-providers-mysql.
However I would like to leave this open for visibility because having dependencies defined without version constraints between providers that might be updated independently could cause things like this to break again in the future due to `pip install` behavior to not update existing modules that already satisfy the requirement.
Because apache-airflow-providers-mysql has no upper version constraint on its apache-airflow-providers-common-sql dependency, any breaking change to a public API in a future version of apache-airflow-providers-common-sql can break existing airflow installs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] MySqlHook constructor fails with exception after upgrading apache-airflow-providers-common-sql from 1.16.0 to 1.17.0 [airflow]
Posted by "rawwar (via GitHub)" <gi...@apache.org>.
rawwar commented on issue #42452:
URL: https://github.com/apache/airflow/issues/42452#issuecomment-2374954401
@tboschtxr , can you use `apache-airflow-providers-postgres==5.13.0` in you requirements.txt?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] MySqlHook constructor fails with exception after upgrading apache-airflow-providers-common-sql from 1.16.0 to 1.17.0 [airflow]
Posted by "steffidlf (via GitHub)" <gi...@apache.org>.
steffidlf commented on issue #42452:
URL: https://github.com/apache/airflow/issues/42452#issuecomment-2373588863
+1, we have the same issue with the same versions
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org