You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "hkc-8010 (via GitHub)" <gi...@apache.org> on 2026/05/06 13:31:23 UTC

[PR] Fix serialized DAG refresh when only bundle version changes [airflow]

hkc-8010 opened a new pull request, #66474:
URL: https://github.com/apache/airflow/pull/66474

   closes: #66301
   
   This fixes the case where `SerializedDagModel.write_dag()` treats a DAG as unchanged
   when the serialized hash matches even though the DAG bundle version has changed.
   
   In that situation, Airflow could skip rewriting serialized metadata and keep stale
   bundle-linked paths around.
   
   This PR updates the refresh logic so that:
   
   - the unchanged-DAG short-circuit only applies when `bundle_name` and `bundle_version`
     both still match
   - `DagCode.fileloc` is refreshed when the file path changes even if the source hash
     stays the same
   - regression coverage verifies both:
     - creating a new `DagVersion` when task instances already exist
     - reusing the same `DagVersion` while refreshing metadata when no task instances exist
   
   Validation:
   - `pytest airflow-core/tests/unit/models/test_serialized_dag.py -q`
   - `breeze testing core-tests --backend postgres --python 3.10 --db-reset -- airflow-core/tests/unit/models/test_serialized_dag.py -q`
   - `breeze testing core-tests --backend postgres --python 3.10 --downgrade-pendulum --db-reset -- airflow-core/tests/unit/models/test_serialized_dag.py -q`
   - `breeze testing core-tests --backend sqlite --python 3.10 --force-lowest-dependencies --db-reset -- airflow-core/tests/unit/models/test_serialized_dag.py -q`
   
   ##### Was generative AI tooling used to co-author this PR?
   
   - [X] Yes (Codex)
   
   Generated-by: Codex following the guidelines
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Fix serialized DAG refresh when only bundle version changes [airflow]

Posted by "hkc-8010 (via GitHub)" <gi...@apache.org>.
hkc-8010 commented on PR #66474:
URL: https://github.com/apache/airflow/pull/66474#issuecomment-4398843374

   @ephraimbuddy Thanks, that makes sense. I agree we shouldn’t rewrite all serialized DAG / DagVersion rows just because the bundle version changes.
   
   I dug further into the callback path, and I think the issue may actually be in dag-processor callback cleanup rather than in `SerializedDagModel`.
   
   What I’m seeing is:
   
   - `DagRun` stores the bundle version at run creation
   - the callback request carries that same bundle version
   - the dag-processor queues a versioned callback file from that request
   - later, removed-file/orphan cleanup seems to compare it against the currently observed files and can drop it even though the same relative file still exists
   
   That’s what `#66484` is addressing by treating `(bundle_name, relative_fileloc)` as the stable identity for presence checks, while still executing the callback against the versioned bundle from the request.
   
   Does that sound like the right direction to you? Or is there still another piece I’m missing here?
   
   More specifically:
   1. Is the expected behavior that a DAG callback should run against the DagRun’s recorded bundle version, even after a newer bundle is deployed?
   2. If yes, does `#66484` cover the right fix, or is there another path I should also be looking at?
   3. If no, what is the intended way for callbacks from older DagRuns to survive a bundle refresh?
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Fix serialized DAG refresh when only bundle version changes [airflow]

Posted by "hkc-8010 (via GitHub)" <gi...@apache.org>.
hkc-8010 commented on PR #66474:
URL: https://github.com/apache/airflow/pull/66474#issuecomment-4399412229

   I dug further into the customer case and re-checked the latest repro. The callback issue happened again even when the relevant DAG was not drifted in \. I also reproduced the current dag-processor behavior directly: a versioned callback file can get treated as orphaned and dropped during cleanup even when the same unversioned DAG file is still present.
   
   That lines up much better with #66484, and I’ve also prepared a customer-side patch from that PR so they can test it on their current Runtime.
   
   I’m going to continue on #66484. Thanks again for the review and the steer here.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Fix serialized DAG refresh when only bundle version changes [airflow]

Posted by "hkc-8010 (via GitHub)" <gi...@apache.org>.
hkc-8010 commented on PR #66474:
URL: https://github.com/apache/airflow/pull/66474#issuecomment-4399410696

   Closing this in favor of #66484, which matches the confirmed callback cleanup bug path more closely.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Fix serialized DAG refresh when only bundle version changes [airflow]

Posted by "hkc-8010 (via GitHub)" <gi...@apache.org>.
hkc-8010 commented on PR #66474:
URL: https://github.com/apache/airflow/pull/66474#issuecomment-4399413207

   I dug further into the customer case and re-checked the latest repro. The callback issue happened again even when the relevant DAG was not drifted in `serdag`. I also reproduced the current dag-processor behavior directly: a versioned callback file can get treated as orphaned and dropped during cleanup even when the same unversioned DAG file is still present.
   
   That lines up much better with #66484, and I’ve also prepared a customer-side patch from that PR so they can test it on their current Runtime.
   
   I’m going to continue on #66484. Thanks again for the review and the steer here.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Fix serialized DAG refresh when only bundle version changes [airflow]

Posted by "hkc-8010 (via GitHub)" <gi...@apache.org>.
hkc-8010 closed pull request #66474: Fix serialized DAG refresh when only bundle version changes
URL: https://github.com/apache/airflow/pull/66474


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Fix serialized DAG refresh when only bundle version changes [airflow]

Posted by "ephraimbuddy (via GitHub)" <gi...@apache.org>.
ephraimbuddy commented on PR #66474:
URL: https://github.com/apache/airflow/pull/66474#issuecomment-4398888000

   > Is the expected behavior that a DAG callback should run against the DagRun’s recorded bundle version, even after a newer bundle is deployed?
   
   Yes
   
   > If yes, does #66484 cover the right fix, or is there another path I should also be looking at?
   
   I believe #66484 will solve it. You can manually confirm as I didn't confirm it manually during the review


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Fix serialized DAG refresh when only bundle version changes [airflow]

Posted by "ephraimbuddy (via GitHub)" <gi...@apache.org>.
ephraimbuddy commented on code in PR #66474:
URL: https://github.com/apache/airflow/pull/66474#discussion_r3202036543


##########
airflow-core/src/airflow/models/serialized_dag.py:
##########
@@ -683,11 +664,8 @@ def write_dag(
             serialized_dag_hash == new_serialized_dag.dag_hash
             and dag_version
             and dag_version.bundle_name == bundle_name
+            and dag_version.bundle_version == bundle_version

Review Comment:
   This was deliberate. We don't want bundle version change to result in rewriting all versions/serdag cc @jedcunningham 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org