(airflow) 02/02: Document pandas 3 impact on DataFrame XComs

[po...@apache.org]

This is an automated email from the ASF dual-hosted git repository.

potiuk pushed a commit to branch pandas-3-support
in repository https://gitbox.apache.org/repos/asf/airflow.git

commit 94ca751f0e31d7d5a0b17c6190ca36e47781b63a
Author: Jarek Potiuk <ja...@potiuk.com>
AuthorDate: Mon Jul 27 18:51:10 2026 +0200

    Document pandas 3 impact on DataFrame XComs
    
    Deployments need to know that every component has to carry the pandas 3
    support before pandas 3 reaches any worker, that a rollback strands the
    XComs written in the meantime, and that a pulled DataFrame now takes its
    dtypes from the reader's pandas version.
---
 airflow-core/newsfragments/70501.significant.rst | 33 ++++++++++++++++++++++++
 1 file changed, 33 insertions(+)

diff --git a/airflow-core/newsfragments/70501.significant.rst b/airflow-core/newsfragments/70501.significant.rst
new file mode 100644
index 00000000000..9dd43544b0d
--- /dev/null
+++ b/airflow-core/newsfragments/70501.significant.rst
@@ -0,0 +1,33 @@
+pandas 3 changes how DataFrame XComs are stored and read back
+
+pandas 3 exposes its public classes from the ``pandas`` namespace, so a DataFrame is qualified as
+``pandas.DataFrame`` instead of ``pandas.core.frame.DataFrame``. XComs record that name alongside the
+serialized value, so the name written into the metadata database depends on the pandas version of the
+component that pushed the value. Airflow registers both names, and a DataFrame written by either
+pandas version can be read by either — no configuration change is needed, and existing XComs stay
+readable.
+
+What you should do:
+
+* **Roll this Airflow version out to every component before pandas 3 reaches any of them** — workers
+  in particular. A component that predates this change cannot read a DataFrame XCom written under
+  pandas 3, and fails the pull with:
+
+  .. code-block:: text
+
+      ImportError: pandas.DataFrame was not found in allow list for deserialization imports.
+      To allow it, add it to allowed_deserialization_classes in the configuration
+
+  The message points at configuration, but the allow list is not the cause and changing it does not
+  help. The rows are not corrupt: they become readable again as soon as the reader is upgraded.
+
+* **Treat a downgrade as a one-way door for those XComs.** Rolling back to an Airflow version without
+  this change strands any DataFrame XCom written while on pandas 3, with the same error, until you
+  roll forward again.
+
+* **Review Dags that inspect the dtypes of a pulled DataFrame.** The pandas version of the *reader*
+  determines what a pulled DataFrame looks like, not the version that wrote it. Under pandas 3, a
+  column of strings comes back with the ``str`` dtype rather than ``object``, and its missing values
+  come back as ``nan`` rather than ``None``. Values are unchanged, but downstream code that branches
+  on ``dtype == "object"``, checks cells with ``is None``, or compares against a reference frame with
+  ``DataFrame.equals()`` can behave differently after the upgrade.