You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "gbengune (via GitHub)" <gi...@apache.org> on 2026/08/22 14:31:40 UTC
[I] pandas dataframe.to_sql on uuid datatype columns [airflow]
gbengune opened a new issue, #71977:
URL: https://github.com/apache/airflow/issues/71977
### Under which category would you file this issue?
Task SDK
### Apache Airflow version
3.3.1
### What happened and how to reproduce it?
Using Airflow docker image for data driven solutions.Immediately I upgraded from 3.3.0 to 3.3.1, I got serious bugs. For instance, using postgresql with sqlalchemy create_engine, I realised while inserting data from a dataframe into a postgresql table with a column with data type uuid, the columns of the dataset coming from the "select" statement of the dataframe are all being handles as strings/text, but in postgresql, you can only explicitly insert a uuid element in a uuid column!So trying to insert the string while casting it on the fly to uuid is not working in Airflow 3.3.1. Basically in all the dags in which I made use of sqlalchemy create_engine in defining the postgresql connection hook failed immediately the code got to the point where I made use of dataframe.to_sql. We would surely have to downgrade back to Airflow 3.3.0 until this bug is solved. Thank you.
### What you think should happen instead?
I think there is a major issue with the pandas's dataframe.to_sql method in Airflow3.3.1 and the datatypes of table columns should remain thesame as the source dataframe. Using string casts on the fly is not going to help while working with special data types.
### Operating System
linus ubuntu noble
### Deployment
None
### Apache Airflow Provider(s)
_No response_
### Versions of Apache Airflow Providers
3.3.1
### Official Helm Chart version
Not Applicable
### Kubernetes Version
_No response_
### Helm Chart configuration
_No response_
### Docker Image customizations
Not applicable
### Anything else?
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] pandas dataframe.to_sql on uuid datatype columns [airflow]
Posted by "hunnguye (via GitHub)" <gi...@apache.org>.
hunnguye commented on issue #71977:
URL: https://github.com/apache/airflow/issues/71977#issuecomment-5510281400
Not fitting to the title but relevant to the root cause is the following unexpected problem which also came with the psycopg update.
```
(psycopg.OperationalError): sending query and params failed: number of parameters must be between 0 and 65535
```
It is mentioned here : https://github.com/psycopg/psycopg/issues/620
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] pandas dataframe.to_sql on uuid datatype columns [airflow]
Posted by "gbengune (via GitHub)" <gi...@apache.org>.
gbengune commented on issue #71977:
URL: https://github.com/apache/airflow/issues/71977#issuecomment-5384995548
@sehajsandhu123 That makes sense considering the fact that I also only tested with different Panda versions and not with sqlalchemy. No wonder all my Dags failed just in pipelines where I used sqlalchemy's create_engine! I also appreciate the new info you dropped considering a new Apache-Airflow feature, which aids in the direct creation of sqlalchemy's DB-connection instead of the previously used URI connection string generator.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] pandas dataframe.to_sql on uuid datatype columns [airflow]
Posted by "boring-cyborg[bot] (via GitHub)" <gi...@apache.org>.
boring-cyborg[bot] commented on issue #71977:
URL: https://github.com/apache/airflow/issues/71977#issuecomment-5380920578
Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] pandas dataframe.to_sql on uuid datatype columns [airflow]
Posted by "sehajsandhu123 (via GitHub)" <gi...@apache.org>.
sehajsandhu123 commented on issue #71977:
URL: https://github.com/apache/airflow/issues/71977#issuecomment-5384799592
@gbengune
I reproduced this and bisected this issue — it's not pandas. The 3.3.1 Docker image bumped apache-airflow-providers-postgres from 6.8.0 to 7.0.x, whose breaking change switches the SQLAlchemy engines built by PostgresHook from psycopg2 to psycopg (v3) whenever SQLAlchemy 2.x is installed.
SQLAlchemy's psycopg (v3) dialect renders typed bind casts, so the insert becomes:
INSERT INTO t (id, ...) VALUES ($1::VARCHAR, ...)
and PostgreSQL refuses the character varying → uuid assignment (with psycopg2, string params are sent untyped and the server coerces them).
I've opened #72000 to fix this: PostgresHook now honors the sqlalchemy_scheme connection extra, so setting {"sqlalchemy_scheme": "postgresql+psycopg2"} on the connection keeps its SQLAlchemy engines on psycopg2 and restores the previous behaviour. The default (psycopg3) is unchanged.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org