[I] Serialized DAG fileloc can remain stale when DAG content hash is unchanged, causing callback/history issues with versioned DAG bundles [airflow]

["hkc-8010 (via GitHub)" <gi...@apache.org>]

hkc-8010 opened a new issue, #66300:
URL: https://github.com/apache/airflow/issues/66300

   ### Apache Airflow version
   
   3.2.0
   
   ### What happened?
   
   We hit a case where a DAG was being actively parsed from the current bundle path, but the serialized DAG metadata still pointed to an older bundle path.
   
   In our case, DAG-level  behavior was failing before a manual serialized-DAG refresh, then started working again after we deleted the serialized row and let Airflow recreate it.
   
   There is still some uncertainty about the exact internal callback execution path, but the stale serialized metadata was definitely present, and refreshing it correlated directly with the fix.
   
   We also saw repeated API/UI-side errors for the same DAG version while this was happening:
   
   
   
   ### What you think should happen instead?
   
   If a DAG's  changes, the serialized DAG row should be updated to reflect the new path even when the serialized DAG content hash is unchanged.
   
   At minimum,  should not stay stale indefinitely when the DAG is being actively reparsed from a new location.
   
   ### How to reproduce
   
   We do not yet have a minimal standalone repro, but this is the observed pattern:
   
   1. Run Airflow with versioned or changing DAG bundle paths.
   2. Keep the DAG definition effectively unchanged so the serialized DAG hash remains the same.
   3. Move the same DAG to a new bundle path.
   4. Observe that  and current parsing reflect the new path, but  may still point to the old path.
   5. DAG-level callback behavior and some UI/version-history lookups may then behave incorrectly.
   
   ### Operating System
   
   Linux
   
   ### Versions of Apache Airflow Providers
   
   Not central to the issue
   
   ### Deployment
   
   Custom / Astro Hosted runtime based on Airflow 3.2.0
   
   ### Deployment details
   
   Observed on Airflow 3.2.0 with versioned DAG bundle paths.
   
   ### Anything else?
   
   A few concrete observations from the incident:
   
   -  was current and pointed at the active bundle path.
   -  was current.
   -  was still old.
   -  still pointed at the old bundle path.
   - The old bundle path no longer existed on the current pod.
   - DAG-level failure callbacks did not appear to fire for failed runs before refresh.
   - After deleting the serialized DAG row and letting Airflow recreate it, the serialized  updated to the current path and the next customer test worked.
   
   The serialization logic appears relevant here. In , the row is skipped when  and  are unchanged:
   
   
   
   But  is set separately from  when building the serialized row:
   
   
   
   So if only  changes while the serialized DAG content hash stays the same, the row may never be rewritten.
   
   ### Are you willing to submit PR?
   
   Yes


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] Serialized DAG fileloc can remain stale when DAG content hash is unchanged, causing callback/history issues with versioned DAG bundles [airflow]

["ephraimbuddy (via GitHub)" <gi...@apache.org>]

ephraimbuddy commented on issue #66300:
URL: https://github.com/apache/airflow/issues/66300#issuecomment-4440610053

   Duplicate of #66301


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] Serialized DAG fileloc can remain stale when DAG content hash is unchanged, causing callback/history issues with versioned DAG bundles [airflow]

["ephraimbuddy (via GitHub)" <gi...@apache.org>]

ephraimbuddy closed issue #66300: Serialized DAG fileloc can remain stale when DAG content hash is unchanged, causing callback/history issues with versioned DAG bundles
URL: https://github.com/apache/airflow/issues/66300


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org