You are viewing a plain text version of this content. The canonical link for it is here.

Posted to commits@airflow.apache.org by "aa-matthias (via GitHub)" <gi...@apache.org> on 2025/07/15 21:05:02 UTC

[D] Constraints in 3.0.3 have incompatible combination of pandas & sqlalchemy [airflow]

GitHub user aa-matthias created a discussion: Constraints in 3.0.3 have incompatible combination of pandas & sqlalchemy

### Apache Airflow version

3.0.3

### If "Other Airflow 2 version" selected, which one?

_No response_

### What happened?

constraints for 3.0.3 have pandas 2.2.3 and SQLAlchemy 1.4.54 causing this issue now:

https://github.com/pandas-dev/pandas/issues/57049

<img width="539" height="29" alt="Image" src="https://github.com/user-attachments/assets/fc14ce86-5abc-40fc-a769-581359a4570d" />

### What you think should happen instead?

_No response_

### How to reproduce

Upgrade to 3.0.3 using constraints for 3.0.3 python3.12

### Operating System

Ubuntu 24.04

### Versions of Apache Airflow Providers

_No response_

### Deployment

Virtualenv installation

### Deployment details

_No response_

### Anything else?

_No response_

### Are you willing to submit PR?

- [ ] Yes I am willing to submit a PR!

### Code of Conduct

- [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)


GitHub link: https://github.com/apache/airflow/discussions/53392

----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org

Re: [D] Constraints in 3.0.3 have incompatible combination of pandas & sqlalchemy [airflow]

Posted by "potiuk (via GitHub)" <gi...@apache.org>.

GitHub user potiuk added a comment to the discussion: Constraints in 3.0.3 have incompatible combination of pandas & sqlalchemy

PR here: https://github.com/apache/airflow/pull/53535

GitHub link: https://github.com/apache/airflow/discussions/53392#discussioncomment-13816711

----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org

Re: [D] Constraints in 3.0.3 have incompatible combination of pandas & sqlalchemy [airflow]

Posted by "jabbera (via GitHub)" <gi...@apache.org>.

GitHub user jabbera added a comment to the discussion: Constraints in 3.0.3 have incompatible combination of pandas & sqlalchemy

@potiuk the airflow constraints file is wrong however: https://[raw.githubusercontent.com/apache/airflow/constraints-main/constraints-3.11.txt](https://raw.githubusercontent.com/apache/airflow/constraints-main/constraints-3.11.txt)

it has: 

pandas==2.2.3
SQLAlchemy==1.4.54

These are not compatible. 

GitHub link: https://github.com/apache/airflow/discussions/53392#discussioncomment-13807935

----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org

Re: [D] Constraints in 3.0.3 have incompatible combination of pandas & sqlalchemy [airflow]

Posted by "potiuk (via GitHub)" <gi...@apache.org>.

GitHub user potiuk added a comment to the discussion: Constraints in 3.0.3 have incompatible combination of pandas & sqlalchemy

Thanks @jabbera for following up with this one despite me being hesitant. It turned out that that our pandas requirement for postgres and common.sql just missed those extras - that was the root of the problem.

Yeah. The pr worked. You will see in "source providers" constraints in main that pandas is now 2.1.4.

https://github.com/apache/airflow/blob/constraints-main/constraints-source-providers-3.11.txt

When the next wave of providers is released (next week) also "constraints-3.11.txt" will be updated (those constraints are build automatically based on the released providers in PyPI (actually in main it should get updated as soon as RCs are released - because our constraint building mechanism includes also RC candidates as of few months).

Then, when we release 3.0.4 it will use the released providers, and then it should get update for 3.0.4 constraints.

Sometimes we fix constraints for past released versions (last time yesterday) - https://airflow.apache.org/docs/docker-stack/changelog.html#changes-after-publishing-the-images -> but we only do it for **really** critical changes that affect many users, in this case I think the issue is really affecting small number of users who want to use from_df/to_df and there is an existing workaround (just downgrade pandas to 2.1.4) so I do not see a need to regenerate the images and update constraints manually.

Unless there are many people complaining and some other reasons why this is problematic to have pandas 2.2.3 in constraints.

GitHub link: https://github.com/apache/airflow/discussions/53392#discussioncomment-13822862

----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org

Re: [D] Constraints in 3.0.3 have incompatible combination of pandas & sqlalchemy [airflow]

Posted by "oailton (via GitHub)" <gi...@apache.org>.

GitHub user oailton added a comment to the discussion: Constraints in 3.0.3 have incompatible combination of pandas & sqlalchemy

I have seen an issue on great-expectations and I guess it could contribute to this discussion. They have dependencies on Pandas, SQLAlchemy and Airflow and one suggestion is a fine tuning on Airflow requirements. I don't know if it is reasonable, my intention was only bring it to your attention.

https://github.com/great-expectations/great_expectations/issues/10621#issuecomment-2990522995

GitHub link: https://github.com/apache/airflow/discussions/53392#discussioncomment-13777244

----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org

Re: [D] Constraints in 3.0.3 have incompatible combination of pandas & sqlalchemy [airflow]

Posted by "aa-matthias (via GitHub)" <gi...@apache.org>.

GitHub user aa-matthias added a comment to the discussion: Constraints in 3.0.3 have incompatible combination of pandas & sqlalchemy

pandas 2.2.3 has the optional requirement for SQLAlchemy >2.0.0

https://github.com/pandas-dev/pandas/blob/0691c5cf90477d3503834d983f69350f250a6ff7/pyproject.toml#L119

however, this workaround seems to work right now: https://github.com/pandas-dev/pandas/issues/57049#issuecomment-2638146451


GitHub link: https://github.com/apache/airflow/discussions/53392#discussioncomment-13772990

----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org

Re: [D] Constraints in 3.0.3 have incompatible combination of pandas & sqlalchemy [airflow]

Posted by "oailton (via GitHub)" <gi...@apache.org>.

GitHub user oailton edited a comment on the discussion: Constraints in 3.0.3 have incompatible combination of pandas & sqlalchemy

I have seen an issue on great-expectations and I guess it could contribute to this discussion. They have dependencies on Pandas, SQLAlchemy and Airflow and one suggestion is a fine tuning on Airflow requirements. I don't know if it is reasonable, my intention was only bring it to your attention.

> A few observations, @billdirks :
> 
> - Pandas 2.2.0+ correctly has a `SQLAlchemy>=2.0.0` requirement, but only for certain db-related extras
> - E.g. `uv add "sqlalchemy<2" "pandas[sql-other]"` will correctly resolve pandas 2.1 and sqlalchemy 1.4
> 
> Currently, to support an Airflow compatibility requirement, we need to support SqlAlchemy 1.4.X.
> 
> So it seems that what is required is that Airflow requires SqlAlchemy 1.4 _and_ the pandas extra "sql-other" and they would be home safe. Or perhaps gx could/should require pandas with that extra. The pin `pandas<2.2` seems a completely wrong way to handle this for sure at least. 

 _Originally posted by @jonaslb in [#10621](https://github.com/great-expectations/great_expectations/issues/10621#issuecomment-2990522995)_

GitHub link: https://github.com/apache/airflow/discussions/53392#discussioncomment-13777244

----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org

Re: [D] Constraints in 3.0.3 have incompatible combination of pandas & sqlalchemy [airflow]

Posted by "jabbera (via GitHub)" <gi...@apache.org>.

GitHub user jabbera edited a comment on the discussion: Constraints in 3.0.3 have incompatible combination of pandas & sqlalchemy

@potiuk the airflow constraints file is wrong however: https://[raw.githubusercontent.com/apache/airflow/constraints-main/constraints-3.11.txt](https://raw.githubusercontent.com/apache/airflow/constraints-main/constraints-3.11.txt)

it has: 

pandas==2.2.3
SQLAlchemy==1.4.54

These are not compatible as pandas >=2.2.0 only supports sqlalchemy>=2.0.0

GitHub link: https://github.com/apache/airflow/discussions/53392#discussioncomment-13807935

----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org

Re: [D] Constraints in 3.0.3 have incompatible combination of pandas & sqlalchemy [airflow]

Posted by "potiuk (via GitHub)" <gi...@apache.org>.

GitHub user potiuk added a comment to the discussion: Constraints in 3.0.3 have incompatible combination of pandas & sqlalchemy

I also converted the discussion back to issue and posted detailed explanation of the problem and explanation on workarounds you might use: https://github.com/apache/airflow/issues/53560 

Let me know if you think this is enough for you: @jacobeulogio @aa-matthias @jabbera and if you are fine with the workarounds posted.

GitHub link: https://github.com/apache/airflow/discussions/53392#discussioncomment-13823115

----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org

Re: [D] Constraints in 3.0.3 have incompatible combination of pandas & sqlalchemy [airflow]

Posted by "jacobeulogio (via GitHub)" <gi...@apache.org>.

GitHub user jacobeulogio added a comment to the discussion: Constraints in 3.0.3 have incompatible combination of pandas & sqlalchemy

Hi, I have the same issue. All tasks that have the df.to_sql() method failed with the same error in airflow v3.0.3. I had to downgrade to 3.0.2 and everything started working again.

GitHub link: https://github.com/apache/airflow/discussions/53392#discussioncomment-13788967

----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org

Re: [D] Constraints in 3.0.3 have incompatible combination of pandas & sqlalchemy [airflow]

Posted by "potiuk (via GitHub)" <gi...@apache.org>.

GitHub user potiuk added a comment to the discussion: Constraints in 3.0.3 have incompatible combination of pandas & sqlalchemy

converting to a discussion as more discussion and input from reporter is needed.

GitHub link: https://github.com/apache/airflow/discussions/53392#discussioncomment-13769422

----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org

Re: [D] Constraints in 3.0.3 have incompatible combination of pandas & sqlalchemy [airflow]

Posted by "jabbera (via GitHub)" <gi...@apache.org>.

GitHub user jabbera edited a comment on the discussion: Constraints in 3.0.3 have incompatible combination of pandas & sqlalchemy

@potiuk the airflow constraints file is wrong however: https://[raw.githubusercontent.com/apache/airflow/constraints-main/constraints-3.11.txt](https://raw.githubusercontent.com/apache/airflow/constraints-main/constraints-3.11.txt)

it has: 

pandas==2.2.3
SQLAlchemy==1.4.54

These are not compatible as pandas >=2.2.0 only supports sqlalchemt>=2.0.0

GitHub link: https://github.com/apache/airflow/discussions/53392#discussioncomment-13807935

----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org

Re: [D] Constraints in 3.0.3 have incompatible combination of pandas & sqlalchemy [airflow]

Posted by "potiuk (via GitHub)" <gi...@apache.org>.

GitHub user potiuk added a comment to the discussion: Constraints in 3.0.3 have incompatible combination of pandas & sqlalchemy

> These are not compatible as pandas >=2.2.0 only supports sqlalchemy>=2.0.0

Technically the two packages **are** compatible when "base" version of those packages are installed.  From what I see - they are only compatible with sqlalchemy 2.2.0 for "postgres", "mysql", "sql-other" and "all" extras

So you can install together `pandas==2.2.0` and `sqlalachemy==1.4.54` . What you can't install together it seems is for example `pandas[postgres]==2.2.0` and `sqlalchemy==1.4.54`.

I see the issue though. I will take a look and see if we should some of the extras in our pandas requirements. That should fix the problem in future versions.

In the meantime feel free to just downgrade pandas. You can also copy the constraints of airlfow, replace pandas version there and use them instead.

GitHub link: https://github.com/apache/airflow/discussions/53392#discussioncomment-13816659

----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org

Re: [D] Constraints in 3.0.3 have incompatible combination of pandas & sqlalchemy [airflow]

Posted by "potiuk (via GitHub)" <gi...@apache.org>.

GitHub user potiuk added a comment to the discussion: Constraints in 3.0.3 have incompatible combination of pandas & sqlalchemy

I am not sur where this error comes from ? How do you think it comes from those version and not your wrong usage? the cohnstraints in airflow are generated when there is no error in unit tests of airflow showing that those versions work. 

As far as I know the issue you have is because some code you are using needs sqlalchemy 2:

https://stackoverflow.com/questions/38332787/pandas-to-sql-to-sqlite-returns-engine-object-has-no-attribute-cursor

Which waits for FAB 5 to be merged: 

https://github.com/apache/airflow/issues/50188

Without knowing what exactly caused your issue, we are unable to spend our time on trying to guess what is causing your problem, but if you can spend some time on providing more details, complete stack traces and more context, we might.

GitHub link: https://github.com/apache/airflow/discussions/53392#discussioncomment-13769421

----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org

Re: [D] Constraints in 3.0.3 have incompatible combination of pandas & sqlalchemy [airflow]

Posted by "potiuk (via GitHub)" <gi...@apache.org>.

GitHub user potiuk edited a comment on the discussion: Constraints in 3.0.3 have incompatible combination of pandas & sqlalchemy

I also converted the discussion back to issue and posted detailed explanation of the problem and explanation on workarounds you might use: https://github.com/apache/airflow/issues/53560 

Let me know if you think this is enough for you: @jacobeulogio @aa-matthias @jabbera @oailton and if you are fine with the workarounds posted.

GitHub link: https://github.com/apache/airflow/discussions/53392#discussioncomment-13823115

----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org

Re: [D] Constraints in 3.0.3 have incompatible combination of pandas & sqlalchemy [airflow]

Posted by "potiuk (via GitHub)" <gi...@apache.org>.

GitHub user potiuk added a comment to the discussion: Constraints in 3.0.3 have incompatible combination of pandas & sqlalchemy

UPDATE: in order to have consistent constraints, you need to update:

* pandas to 2.1.4
* numpy to 1.26.4

Otherwise numpy will generate conflict. I updated the workaround description in the issue.

GitHub link: https://github.com/apache/airflow/discussions/53392#discussioncomment-13823134

----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org

Re: [D] Constraints in 3.0.3 have incompatible combination of pandas & sqlalchemy [airflow]

Posted by "potiuk (via GitHub)" <gi...@apache.org>.

GitHub user potiuk edited a comment on the discussion: Constraints in 3.0.3 have incompatible combination of pandas & sqlalchemy

Thanks @jabbera for following up with this one despite me being hesitant. It turned out that that our pandas requirement for postgres and common.sql providers just missed those extras - that was the root of the problem.

Yeah. The pr worked. You will see in "source providers" constraints in main that pandas is now 2.1.4.

https://github.com/apache/airflow/blob/constraints-main/constraints-source-providers-3.11.txt

Then, when we release 3.0.4 it will use the released providers, and then it should get update for 3.0.4 constraints.

Unless there are many people complaining and some other reasons why this is problematic to have pandas 2.2.3 in constraints.

GitHub link: https://github.com/apache/airflow/discussions/53392#discussioncomment-13822862

----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org

Re: [D] Constraints in 3.0.3 have incompatible combination of pandas & sqlalchemy [airflow]

Posted by "oailton (via GitHub)" <gi...@apache.org>.

GitHub user oailton edited a comment on the discussion: Constraints in 3.0.3 have incompatible combination of pandas & sqlalchemy

I have seen an issue on great-expectations and I guess it could contribute to this discussion. They have dependencies on Pandas, SQLAlchemy and Airflow and one suggestion is a fine tuning on Airflow requirements. I don't know if it is reasonable, my intention was only to bring it to your attention.

> A few observations, @billdirks :
> 
> - Pandas 2.2.0+ correctly has a `SQLAlchemy>=2.0.0` requirement, but only for certain db-related extras
> - E.g. `uv add "sqlalchemy<2" "pandas[sql-other]"` will correctly resolve pandas 2.1 and sqlalchemy 1.4
> 
> Currently, to support an Airflow compatibility requirement, we need to support SqlAlchemy 1.4.X.
> 
> So it seems that what is required is that Airflow requires SqlAlchemy 1.4 _and_ the pandas extra "sql-other" and they would be home safe. Or perhaps gx could/should require pandas with that extra. The pin `pandas<2.2` seems a completely wrong way to handle this for sure at least. 

 _Originally commented by @jonaslb in [great-expectations issue #10621](https://github.com/great-expectations/great_expectations/issues/10621#issuecomment-2990522995)_

GitHub link: https://github.com/apache/airflow/discussions/53392#discussioncomment-13777244

----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org