You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "aa-matthias (via GitHub)" <gi...@apache.org> on 2024/02/27 10:06:19 UTC
[I] SQLAlchemy / Pandas incompatibility [airflow]
aa-matthias opened a new issue, #37742:
URL: https://github.com/apache/airflow/issues/37742
### Apache Airflow version
2.8.2
### If "Other Airflow 2 version" selected, which one?
_No response_
### What happened?
After upgrading to Airflow 2.8.2 one of our DAGs started failing. It seems to come from updating Pandas to 2.2.1 while SQLAlchemy is still on 1.4.X. According to Pandas docs v2.2.1 needs at least 2.0.
https://pandas.pydata.org/docs/whatsnew/v2.2.0.html#increased-minimum-versions-for-dependencies
The issue we get is from within pandas
File "./virtualenvs/airflow/lib/python3.11/site-packages/pandas/io/sql.py", line 2670, in execute
cur = self.con.cursor()
^^^^^^^^^^^^^^^
AttributeError: 'Connection' object has no attribute 'cursor'
### What you think should happen instead?
_No response_
### How to reproduce
Upgrade to 2.8.2 using the Constraint File for Python 3.11
### Operating System
Ubuntu 22.04.4 LTS
### Versions of Apache Airflow Providers
_No response_
### Deployment
Virtualenv installation
### Deployment details
_No response_
### Anything else?
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] SQLAlchemy / Pandas incompatibility [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk closed issue #37742: SQLAlchemy / Pandas incompatibility
URL: https://github.com/apache/airflow/issues/37742
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] SQLAlchemy / Pandas incompatibility [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on issue #37742:
URL: https://github.com/apache/airflow/issues/37742#issuecomment-3145191731
> The constraint is no more
Not sure what it means
> Breaking pandas x SQLAlchemy in Airflow 3 right now.
You can upgrade things independently from constraints. But this has been fixed in https://github.com/apache/airflow/pull/53535 and new constraints are coming in 3.0.4 - right after we release current wave of providers that is being voted.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] SQLAlchemy / Pandas incompatibility [airflow]
Posted by "TopCoder2K (via GitHub)" <gi...@apache.org>.
TopCoder2K commented on issue #37742:
URL: https://github.com/apache/airflow/issues/37742#issuecomment-3958287740
@potiuk, what pandas version should be used with Airflow 3.1.7 (latest) and Python 3.12? I tried the one from constraints (https://raw.githubusercontent.com/apache/airflow/constraints-3.1.7/constraints-3.12.txt), i.d. `pandas=2.3.3`, but had the same error (`AttributeError: 'Connection' object has no attribute 'cursor'`).
Everything works with `pandas=2.1.4` 🤔, although I'm afraid that this version is old.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] SQLAlchemy / Pandas incompatibility [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on issue #37742:
URL: https://github.com/apache/airflow/issues/37742#issuecomment-3958321672
Likely you have some sqlalchemy version that conflicts
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] SQLAlchemy / Pandas incompatibility [airflow]
Posted by "aa-matthias (via GitHub)" <gi...@apache.org>.
aa-matthias commented on issue #37742:
URL: https://github.com/apache/airflow/issues/37742#issuecomment-1990929207
@potiuk - looks like the issue reappeared in Airflow 2.8.3 ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] SQLAlchemy / Pandas incompatibility [airflow]
Posted by "Taragolis (via GitHub)" <gi...@apache.org>.
Taragolis commented on issue #37742:
URL: https://github.com/apache/airflow/issues/37742#issuecomment-1991348110
> Can you please elaborate how it reappeared? What makes you think so ?
We exclude https://github.com/apache/airflow/pull/37748 from 2.8.3 and originally I did not add this tests https://github.com/apache/airflow/pull/37779 as 2.8.3 milestone `¯\_(ツ)_/¯`.
The things happen, just need to think how we might prevent this into the future.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] SQLAlchemy / Pandas incompatibility [airflow]
Posted by "lucasfcnunes (via GitHub)" <gi...@apache.org>.
lucasfcnunes commented on issue #37742:
URL: https://github.com/apache/airflow/issues/37742#issuecomment-3143918133
@potiuk
The constraint is no more and it's breaking pandas x SQLAlchemy in Airflow 3 right now.
```console
$ docker run --rm --pull=always --entrypoint='' 'apache/airflow:3.0.3-python3.12' bash -c 'pip list --format=freeze' | grep -Ei '^(pandas|sqlalchemy)=='`
3.0.3-python3.12: Pulling from apache/airflow
Digest: sha256:66d35503f78e8c87de2db43f0a3311efd26a5d34584599509a8bd90fb308adc7
Status: Image is up to date for apache/airflow:3.0.3-python3.12
pandas==2.2.3
SQLAlchemy==1.4.54
```
PS: When `SQLAlchemy<2`, `pandas<2.2` is a must.
Refs:
- https://github.com/pandas-dev/pandas/issues/57049
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] SQLAlchemy / Pandas incompatibility [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on issue #37742:
URL: https://github.com/apache/airflow/issues/37742#issuecomment-1991380338
Let me see - I can check and regenerate the images again. I think the way to do it in the future is to "remember" :) . There is no other way
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] SQLAlchemy / Pandas incompatibility [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on issue #37742:
URL: https://github.com/apache/airflow/issues/37742#issuecomment-3958660486
No not really, the constraints are only active in the command they are used. This is how constraints work. If you do uv add or whatever without constraints they can feely upgrade or downgrade other deps. And even If you specify constraints and install only one or two packages - those constraints only apply to things being explicitly installed, if uv decides to downgrade packages that are not specified in the installation command, they do not follow constraints. If you notice that uv downgrades some package with add and you want to keep it, you need to add this package explicitly in your add command I guess
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] SQLAlchemy / Pandas incompatibility [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on issue #37742:
URL: https://github.com/apache/airflow/issues/37742#issuecomment-1991119234
> @potiuk - looks like the issue reappeared in Airflow 2.8.3 ?
Can you please elaborate how it reappeared? What makes you think so ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] SQLAlchemy / Pandas incompatibility [airflow]
Posted by "TopCoder2K (via GitHub)" <gi...@apache.org>.
TopCoder2K commented on issue #37742:
URL: https://github.com/apache/airflow/issues/37742#issuecomment-3958604537
@potiuk, thank you for the fast answer!
> Did you install everything with constraints ?
Yes, I did, but I used `uv add` instead of `uv pip install` (as the installation guide recommends) since the latter doesn't update `pyproject.toml` which I need to maintain for running tests. I noticed that the constraints state `SQLAlchemy==2.0.46`, whereas I have `sqlalchemy==1.4.54` in my test env. So, `uv add` doesn't work correctly for some reason 😢
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] SQLAlchemy / Pandas incompatibility [airflow]
Posted by "lucasfcnunes (via GitHub)" <gi...@apache.org>.
lucasfcnunes commented on issue #37742:
URL: https://github.com/apache/airflow/issues/37742#issuecomment-3146713312
> > The constraint is no more
>
> Not sure what it means
I meant this PR here https://github.com/apache/airflow/pull/37748
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] SQLAlchemy / Pandas incompatibility [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on issue #37742:
URL: https://github.com/apache/airflow/issues/37742#issuecomment-1967445035
We updated constraints, and re-built the images. When users will be using reference image or install airflow 2.8.2 following our recommendation (with constraints https://airflow.apache.org/docs/apache-airflow/stable/installation/installing-from-pypi.html ) - they should get Pandas 2.1.4 when installing Airflow 2.8.2. Thansk @aa-matthias for notifying us! 1
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] SQLAlchemy / Pandas incompatibility [airflow]
Posted by "aa-matthias (via GitHub)" <gi...@apache.org>.
aa-matthias commented on issue #37742:
URL: https://github.com/apache/airflow/issues/37742#issuecomment-1966199836
https://github.com/pandas-dev/pandas/issues/57049
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] SQLAlchemy / Pandas incompatibility [airflow]
Posted by "Taragolis (via GitHub)" <gi...@apache.org>.
Taragolis commented on issue #37742:
URL: https://github.com/apache/airflow/issues/37742#issuecomment-1991351388
So for Python > 3.8 we generate constraints, build docker image with pandas greater or equal than 2.2
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] SQLAlchemy / Pandas incompatibility [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on issue #37742:
URL: https://github.com/apache/airflow/issues/37742#issuecomment-3148526373
BTW. The new providers are released - if you - for example try to install latest common.sql provider - it will not allow to upgrade pandas beyond what is supported.
The constraints-3-0 generated today (and those will be used for 3.0.4) contain this:
<img width="1472" height="132" alt="Image" src="https://github.com/user-attachments/assets/6260ab38-293c-4530-b530-107f8f4972c3" />
As you can see - pandas is limited in constraints to 2.1.4 now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] SQLAlchemy / Pandas incompatibility [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on issue #37742:
URL: https://github.com/apache/airflow/issues/37742#issuecomment-3958318820
Did you install **everything** with constraints ? Including providers and other reps or you installed things independently ? If independently then sometimes it might be trial and error because some dependencies might not work with others. I recommend to install airflow following our installation guide (with everything you need with constraints) and then downgrade or upgrade any other dependencies as needed and test. This is exactly what our recommendations at installation page are. If you do it differently, there are too many variables to help you and give you concrete answers for potential incompatibilities between other dependencies
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org