You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "potiuk (via GitHub)" <gi...@apache.org> on 2026/07/27 19:03:16 UTC
[PR] Support pandas 3 for DataFrame XComs [airflow]
potiuk opened a new pull request, #70558:
URL: https://github.com/apache/airflow/pull/70558
pandas 3 exposes its public classes from the `pandas` namespace, so a DataFrame is qualified as `pandas.DataFrame` rather than `pandas.core.frame.DataFrame`. The serde registry is keyed on that name and only knew the old one, so under pandas 3 **no DataFrame could be pushed through XCom at all**:
```
TypeError: cannot serialize object of type <class 'pandas.DataFrame'>
```
Both names are now registered, so a DataFrame written by either pandas version can be read by either. Verified across all four combinations (writer x reader), with the payload written by one interpreter and read by the other:
| Reader | pandas-2 payload | pandas-3 payload |
|---|---|---|
| pandas 2.3.3 | `object`, missing = `None` | `object`, missing = `None` |
| pandas 3.0.5 | `str`, missing = `nan` | `str`, missing = `nan` |
The reader's pandas version decides the dtypes, not the writer's — that, and the fact that a component without this change cannot read a pandas-3-written DataFrame XCom, is what the significant newsfragment documents.
Two provider tests also asserted on pandas 2 behaviour (`object` dtype, missing values stringified to `"nan"` / `"None"`) and are now version-aware. The production paths were already correct.
Split out of #70501, which holds the development environment on pandas 2 in the meantime.
---
##### Was generative AI tooling used to co-author this PR?
- [X] Yes — Claude Code (Opus 5)
Generated-by: Claude Code (Opus 5) following [the guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Support pandas 3 for DataFrame XComs [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #70558:
URL: https://github.com/apache/airflow/pull/70558#issuecomment-5128792966
Hey @vatsrahul1001 -> I think that one might be useful to get into 3.3.1 as well. Some of the recent upgrades in our providers "freed" pandas to be updated to version 3 in a number of combinations - and if someone uses Pandas 2 directly sent as XCom, the XComs that are stored in DB will start failing tasks reading them if the tasks will get Pandas 3.
This is **easily** fixable - by this one - the only issue is that there is no way back after migrating to Pandas 3 and that you have to migrate everything at once to Pandas 3 (but this is the only migration scenario we support).
That's why the significant newsfragment.
We could wait to Airflow 3.4, but I am afraid, people will start to migrate to Pandas 3 whenever other Airflow dependencies allow them to, so there is a high risk between now and 3.4 lack of this will hit real users.
We could - alternatively - limit 3.3.1 to "pandas<3" - but that's only half the solution, because older versions of Airflow do not have "pandas<3" limitation - and if someone updates providers **and** updates pandas to 3 - they will have no easy way back. By releasing Airflow 3.3.1 with pandas 3 support, we give them the option "Upgrade to Airlfow 3.3.1" if this happens.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Support pandas 3 for DataFrame XComs [airflow]
Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh commented on PR #70558:
URL: https://github.com/apache/airflow/pull/70558#issuecomment-5180477053
Superseded by: https://github.com/apache/airflow/pull/71103
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Support pandas 3 for DataFrame XComs [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk closed pull request #70558: Support pandas 3 for DataFrame XComs
URL: https://github.com/apache/airflow/pull/70558
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Support pandas 3 for DataFrame XComs [airflow]
Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh commented on code in PR #70558:
URL: https://github.com/apache/airflow/pull/70558#discussion_r3712507895
##########
task-sdk/tests/task_sdk/serde/test_serializers.py:
##########
@@ -289,12 +289,18 @@ def test_pandas_serializers(self):
@pytest.mark.parametrize(
("klass", "version", "data", "msg"),
[
- (pd.DataFrame, 999, "", r"serialized 999 of pandas.core.frame.DataFrame > 1"), # version too new
+ # pandas 3 qualifies the class as pandas.DataFrame, pandas 2 as pandas.core.frame.DataFrame
+ (
+ pd.DataFrame,
+ 999,
+ "",
+ r"serialized 999 of pandas(\.core\.frame)?\.DataFrame > 1",
+ ), # version too new
Review Comment:
The round-trip test and the error message tests only exercise whichever pandas major is installed.
Maybe we should add a test that manufactures an encoded payload with the older version too.(eg: `pandas.core.frame.DataFrame` as CLASSNAME and assert deserialize() succeeds while running under pandas 3, and the reverse).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org