[PR] Prevent deadlocks when Dags update shared assets [airflow]

["ephraimbuddy (via GitHub)" <gi...@apache.org>]

ephraimbuddy opened a new pull request, #71936:
URL: https://github.com/apache/airflow/pull/71936

   Assets are shared across Dags and bundles by design, so the same rows are reached by more than one writer -- two Dag processors, or one alongside ``airflow dags reserialize``. The order they were collected in was the order the Dags happened to arrive, and that order is the order the rows are inserted and so the order they are locked, which is all it takes for two writers to deadlock on assets neither of them conflicts over.
   
   Sorting after collecting rather than while, so a repeated asset is still decided by the last definition of it rather than by where it sorts. Sorted again after the existing rows are read back, since the query returns those in whatever order it likes and activation locks them in the order it is handed.
   
   This orders assets by identity. It does not order the other way ``asset_active`` can conflict, on a uri claimed under two names, which no ordering can fix.
   
    <!-- SPDX-License-Identifier: Apache-2.0
         https://www.apache.org/licenses/LICENSE-2.0 -->
   
   <!--
   Thank you for contributing!
   
   Please provide above a brief description of the changes made in this pull request.
   Write a good git commit message following this guide: https://chris.beams.io/posts/git-commit/
   
   Please make sure that your code changes are covered with tests.
   And in case of new features or big changes remember to adjust the documentation.
   
   For user-facing UI changes, please attach before/after screenshots (or a short
   screen recording) so reviewers can assess the visual impact.
   
   Feel free to ping (in general) for the review if you do not see reaction for a few days
   (72 Hours is the minimum reaction time you can expect from volunteers) - we sometimes miss notifications.
   
   In case of an existing issue, reference it using one of the following:
   
   * closes: #ISSUE
   * related: #ISSUE
   -->
   
   ---
   
   ##### Was generative AI tooling used to co-author this PR?
   
   <!--
   If generative AI tooling has been used in the process of authoring this PR, please
   change below checkbox to `[X]` followed by the name of the tool, uncomment the "Generated-by".
   -->
   
   - [x] Yes (please specify the tool below)
   
   Generated-by: Claude Opus 5 following [the guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] Prevent deadlocks when Dags update shared assets [airflow]

["jedcunningham (via GitHub)" <gi...@apache.org>]

jedcunningham commented on code in PR #71936:
URL: https://github.com/apache/airflow/pull/71936#discussion_r3857228581


##########
airflow-core/src/airflow/dag_processing/collection.py:
##########
@@ -923,8 +941,21 @@ def activate_assets_if_possible(self, models: Iterable[AssetModel], *, session:
             from sqlalchemy.dialects.sqlite import insert as sqlite_insert
 
             stmt = sqlite_insert(AssetActive).on_conflict_do_nothing()
-        if values := [{"name": m.name, "uri": m.uri} for m in models]:
-            session.execute(stmt, values)
+        # Of two assets sharing a name or a uri only the one offered first is activated. Choosing
+        # that here rather than leaving it to the insert means the rows can then be sorted, which
+        # they must be: ``asset_active`` is unique on both columns, so two writers inserting them
+        # in opposite orders deadlock on the index.
+        claimed_names: set[str] = set()
+        claimed_uris: set[str] = set()
+        values = []
+        for model in models:
+            if model.name in claimed_names or model.uri in claimed_uris:
+                continue

Review Comment:
   These two sets only know about the current batch, so an asset that's going to get rejected by the insert anyway can still grab the claim and knock out one that would've made it in.
   
   Say asset_active already has `("n9", "s3://u1")`, and this batch offers `("n1", "s3://u1")` then `("n1", "s3://u2")`. Before, the second one went in as its name and uri were both free. Now the first grabs n1, the second gets skipped, and then the first's insert conflicts anyway, so we end up with neither.
   
   The scheduler's `_activate_referenced_assets` does it the other way round - it looks at what's already active and skips before recording the claim. Seeding these two sets from the existing asset_active rows would match that, and the sorted insert still works.



##########
airflow-core/src/airflow/dag_processing/collection.py:
##########
@@ -850,6 +855,10 @@ def collect(cls, dags: dict[str, LazyDeserializedDAG]) -> Self:
                 dag_id: list(_get_dag_assets(dag, SerializedAsset, inlets=False, outlets=True))
                 for dag_id, dag in dags.items()
             },
+            # Left in the order the Dags define them: that order decides which of two assets

Review Comment:
   The read-back in sync_assets isn't ordered, so this only holds when all the assets are new. Anything already in asset from a previous parse comes back in DB order and gets offered first. Same as before this PR, so not a regression, but the comment and the new test both read as though collection order always wins, and the test clears assets each time so it only ever hits the all-new case.
   
   If you want the claim to actually hold, rebuilding the dict in self.assets order at the end would do it (`return {key: orm_assets[key] for key in self.assets if key in orm_assets}`), and then the test could cover the existing-asset half too. Happy to leave it if you'd rather keep the scope tight, in that case maybe just narrow the comment.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org