You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "ephraimbuddy (via GitHub)" <gi...@apache.org> on 2026/07/14 10:39:57 UTC
[PR] Add serialized Dag version diff engine [airflow]
ephraimbuddy opened a new pull request, #69864:
URL: https://github.com/apache/airflow/pull/69864
Adds the shared engine that computes an observed-state diff between two stored serialized Dag versions. Dag authors and operators need a reliable way to understand how two stored versions differ without downloading serialized payloads or manually comparing implementation details.
This is the **foundation** of a three-part stack; the REST API endpoint and the `airflow dags versions diff` CLI command follow in separate PRs that build on this engine. It intentionally ships with no runtime caller yet — it is fully exercised by unit tests.
### What it does
- `build_serialized_dag_diff` — pure, deterministic, bounded diff over two serialized payloads. Normalizes task/schedule/dependency/metadata state, converts legacy schema shapes (v1/v2 → v3), keys order-insensitive collections, categorizes each change (task, dependency, schedule, param, asset, callback, deadline, metadata, provenance) and its impact, and emits digests plus optional raw values.
- `get_dag_version_diff` — DB-facing wrapper that loads two `DagVersion`s, assembles provenance, and classifies optional source/values data as available, redacted, or unavailable.
- Malformed legacy payloads and missing historical data degrade to an `unavailable` result instead of failing the whole request.
### Testing
- 32 unit tests, including a `@pytest.mark.db_test` class covering `get_dag_version_diff` end-to-end: observed-state diff, not-found handling, version-number validation, and the values/source status plumbing (available, redacted, unavailable-when-code-missing).
- prek (ruff, mypy, license, no-new-AirflowException) clean.
No newsfragment: the engine has no user-facing surface on its own; user-facing notes ship with the API and CLI PRs.
---
##### Was generative AI tooling used to co-author this PR?
- [X] Yes — Claude Code (Opus 4.8)
Generated-by: Claude Code (Opus 4.8) following [the guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions)
---
Drafted-by: Claude Code (Opus 4.8) (no human review before posting)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add serialized Dag version diff engine [airflow]
Posted by "ephraimbuddy (via GitHub)" <gi...@apache.org>.
ephraimbuddy commented on PR #69864:
URL: https://github.com/apache/airflow/pull/69864#issuecomment-5623811388
> One note that doesn't fit inline: the description says the diff engine "is fully exercised by unit tests." The precedence issue I flagged in `_canonicalize_payload_v1` got through because the mapped-task default tests only shadow client defaults via `partial_kwargs`. No case sets a partial-only field at the task's _top level_, so that resolution path has no coverage at all. The one-line fix I tried passed all 186 existing tests, which is how I noticed the gap rather than the bug.
>
> Probably worth a parametrized case that sets `retries`/`pool` at the top level with a conflicting `client_defaults.tasks` value, and softening the claim in the description.
Resolved alongside other reviews
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add serialized Dag version diff engine [airflow]
Posted by "kaxil (via GitHub)" <gi...@apache.org>.
kaxil commented on PR #69864:
URL: https://github.com/apache/airflow/pull/69864#issuecomment-5706905209
Everything from my last round is addressed. I re-checked the three open comments at `733cfaa8` myself, so here is the whole set in one place with what it would take to close it out.
**One thing holds my approval: the counter at line 500.**
I applied the one-line change (move `self.count += 1` below the repeat early-return) and measured both sides in breeze at this HEAD. Redacted mode, a Dag where every task changes `retries` plus one `/dag/timezone` change:
| case | today | with the counter moved |
|---|---|---|
| 600 tasks, `max_changes=500` (the default) | `truncated: true`, 1 record, timezone absent | `truncated: false`, both paths, exact counts |
| 5000 tasks, `max_changes=5000` (`MAX_ALLOWED_CHANGES`) | `truncated: true`, 1 record, timezone absent | `truncated: false`, both paths, exact counts |
At the default bound a 600-task fan-out already hides the timezone change, and at 5000 tasks nothing a caller may pass recovers it, because 5000 is the ceiling. So for "bumped `retries` in `default_args` and changed the schedule" on a large Dag, the operator sees one record saying retries changed on `*` tasks plus `truncated: true`, and the schedule change is gone with nothing pointing at it. That is the shape the endpoint PR freezes.
The change costs one test parametrization. Engine plus model suites with it applied: `1 failed, 621 passed`, and the failure is `test_build_diff_excludes_new_group_beyond_change_limit[False-2]`, which under the change returns both records and `truncated: false`.
I think that test is the one to move, because the suite currently pins both answers. `test_build_diff_bound_withholds_records_not_public_paths` asserts `redacted["truncated"] is False` at `max_changes=1` for ten occurrences of a single path, with the comment "no bound can withhold a path from the caller or cost an occurrence". `test_build_diff_excludes_new_group_beyond_change_limit[False-2]` is that same situation with a second public path present, and there the bound does withhold one. Whether a later distinct path survives depends on how many occurrences the earlier paths happened to have, even though those occurrences produced no records.
Keeping the current counting is a reasonable call to make instead. In that case the docstring at lines 112-124 and the name of `test_build_diff_bound_withholds_records_not_public_paths` both need to say that a bound can withhold a public path, since today they say it cannot.
**The other two comments are optional and will not hold approval.** The canonicalization handler logging only `error_type` costs you the `unsupported client_defaults sections: [...]` message you added two rounds ago, and the unreachable `serialized_dag_recursion_limit_exceeded` is a wire-contract question the CLI and endpoint PRs will inherit from line 53. Take them or leave them.
Nothing else from me. I checked the reason list against every `_mark_unavailable` call site and the two `get_diff` adds (complete and accurate apart from the unreachable one), `get_diff` itself (one query with `joinedload`/`selectinload`, bound validated before the lookup, no writes), and the `serialized_objects.py` refactor (`_DAG_CALLBACK_FIELDS` iteration order cannot reach `dag_hash` or the stored blob, both of which go through `json.dumps(..., sort_keys=True)`). Settle the counter either way and I approve.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add serialized Dag version diff engine [airflow]
Posted by "ephraimbuddy (via GitHub)" <gi...@apache.org>.
ephraimbuddy commented on PR #69864:
URL: https://github.com/apache/airflow/pull/69864#issuecomment-5772536682
Good to merge @kaxil ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add serialized Dag version diff engine [airflow]
Posted by "ephraimbuddy (via GitHub)" <gi...@apache.org>.
ephraimbuddy commented on PR #69864:
URL: https://github.com/apache/airflow/pull/69864#issuecomment-5631201995
One note here @kaxil @potiuk , I removed source diff since that is already in the UI. This is now only the serialized diff
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add serialized Dag version diff engine [airflow]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #69864:
URL: https://github.com/apache/airflow/pull/69864#discussion_r3580125690
##########
airflow-core/src/airflow/models/dag_version.py:
##########
@@ -243,6 +251,126 @@ def version(self) -> str:
"""A human-friendly representation of the version."""
return f"{self.dag_id}-{self.version_number}"
+ @classmethod
+ @provide_session
+ def get_diff(
+ cls,
+ dag_id: str,
+ base_version_number: int,
+ target_version_number: int,
+ *,
+ include_values: bool = False,
+ include_source: bool = False,
+ max_changes: int | None = None,
+ source_status: SourceStatus | None = None,
+ values_status: ValuesStatus | None = None,
+ session: Session = NEW_SESSION,
+ ) -> dict[str, Any]:
+ """
+ Compare two versions of a Dag using their currently stored state.
+
+ ``source_status`` is supplied by callers that have an authorization context. The CLI has
+ operator-level authority and leaves it unset, while API callers calculate it before this
+ method returns any source content. API callers use ``values_status`` to suppress raw values
+ when the user cannot access Dag code; the structural diff remains available in redacted form.
+ """
+ from airflow.serialization.dag_version_diff import (
+ DEFAULT_MAX_CHANGES,
+ build_serialized_dag_diff,
+ )
Review Comment:
The function-local import here is presumably to avoid a circular import (DagVersion → dag_version_diff → DagSerialization → SerializedDAG definition → DagVersion). Adding a short comment explaining the reason helps future refactors avoid “fixing” it back to a module import and reintroducing the cycle.
##########
airflow-core/src/airflow/models/dag_version.py:
##########
@@ -243,6 +251,126 @@ def version(self) -> str:
"""A human-friendly representation of the version."""
return f"{self.dag_id}-{self.version_number}"
+ @classmethod
+ @provide_session
+ def get_diff(
+ cls,
+ dag_id: str,
+ base_version_number: int,
+ target_version_number: int,
+ *,
+ include_values: bool = False,
+ include_source: bool = False,
+ max_changes: int | None = None,
+ source_status: SourceStatus | None = None,
+ values_status: ValuesStatus | None = None,
+ session: Session = NEW_SESSION,
+ ) -> dict[str, Any]:
+ """
+ Compare two versions of a Dag using their currently stored state.
+
+ ``source_status`` is supplied by callers that have an authorization context. The CLI has
+ operator-level authority and leaves it unset, while API callers calculate it before this
+ method returns any source content. API callers use ``values_status`` to suppress raw values
+ when the user cannot access Dag code; the structural diff remains available in redacted form.
+ """
+ from airflow.serialization.dag_version_diff import (
+ DEFAULT_MAX_CHANGES,
+ build_serialized_dag_diff,
+ )
+
+ if max_changes is None:
+ max_changes = DEFAULT_MAX_CHANGES
+ if base_version_number < 1 or target_version_number < 1:
+ raise ValueError("Dag version numbers must be positive integers")
+
Review Comment:
`values_status` / `source_status` are runtime strings and currently aren’t validated. A mistyped/incorrect `values_status` (e.g. "redacted") would be treated as allowed and could unintentionally enable returning raw values when `include_values=True`. Validating these parameters keeps the response schema stable and avoids accidental data exposure caused by caller bugs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add serialized Dag version diff engine [airflow]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #69864:
URL: https://github.com/apache/airflow/pull/69864#discussion_r3580125586
##########
airflow-core/src/airflow/models/dag_version.py:
##########
@@ -40,6 +41,13 @@
log = logging.getLogger(__name__)
+SourceStatus = Literal["current_stored_code", "redacted", "unavailable"]
+ValuesStatus = Literal["available", "unavailable"]
+
+
+class DagVersionNotFoundError(ValueError):
+ """Raised when one of the requested Dag versions does not exist."""
+
Review Comment:
`DagVersionNotFoundError` is very close in meaning/name to the existing `airflow.exceptions.DagVersionNotFound` (used in other parts of the codebase). Having two similarly named exceptions for “DagVersion missing” increases ambiguity for callers and error handling (which one should API/CLI layer catch/translate?). Consider reusing `DagVersionNotFound` or renaming this exception to be more specific to this method (and updating tests accordingly).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add serialized Dag version diff engine [airflow]
Posted by "ephraimbuddy (via GitHub)" <gi...@apache.org>.
ephraimbuddy commented on code in PR #69864:
URL: https://github.com/apache/airflow/pull/69864#discussion_r4005500872
##########
airflow-core/src/airflow/serialization/dag_version_diff.py:
##########
@@ -0,0 +1,896 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""Observed-state diffs for serialized Dag payloads."""
+
+from __future__ import annotations
+
+import copy
+import hashlib
+import json
+from collections.abc import Callable, Mapping
+from datetime import timedelta
+from enum import Enum
+from typing import Any, Literal
+
+import structlog
+
+from airflow.serialization.definitions.baseoperator import SerializedBaseOperator
+from airflow.serialization.definitions.mappedoperator import SerializedMappedOperator
+from airflow.serialization.serialized_objects import (
+ _DAG_CALLBACK_FIELDS,
+ _OPERATOR_TIMEDELTA_FIELDS,
+ DagSerialization,
+ OperatorSerialization,
+)
+
+log = structlog.get_logger(__name__)
+
+DIFF_SCHEMA_VERSION = 1
+DEFAULT_MAX_CHANGES = 500
+MAX_ALLOWED_CHANGES = 5000
+SUPPORTED_SERIALIZED_DAG_SCHEMA_VERSIONS = frozenset((1, 2, 3))
+
+_ORDER_INSENSITIVE_LIST_PATHS = {
+ ("dag", "tags"),
+ ("dag", "allowed_run_types"),
+}
+_KEYED_COLLECTION_PATHS = {
+ ("dag", "tasks"),
+ ("dag", "dag_dependencies"),
+ *_ORDER_INSENSITIVE_LIST_PATHS,
+}
+_CUSTOM_TASK_FIELDS_PATH_COMPONENT = "custom_fields"
+# This allowlist is part of diff schema v1. Serializer schema changes must not
+# silently change the paths visible to callers of the diff API.
+_DIFF_V1_PUBLIC_TASK_FIELDS = frozenset(
+ {
+ "__type",
+ "_disallow_kwargs_override",
+ "_expand_input_attr",
+ "_is_mapped",
+ "_is_sensor",
+ "_logger_name",
+ "_needs_expansion",
+ "_operator_extra_links",
+ "_task_display_name",
+ "_task_module",
+ "allow_nested_operators",
+ "depends_on_past",
+ "do_xcom_push",
+ "doc",
+ "doc_json",
+ "doc_md",
+ "doc_rst",
+ "doc_yaml",
+ "downstream_task_ids",
+ "email_on_failure",
+ "email_on_retry",
+ "end_date",
+ "execution_timeout",
+ "executor",
+ "executor_config",
+ "has_on_execute_callback",
+ "has_on_failure_callback",
+ "has_on_retry_callback",
+ "has_on_skipped_callback",
+ "has_on_success_callback",
+ "ignore_first_depends_on_past",
+ "inlets",
+ "is_setup",
+ "is_teardown",
+ "map_index_template",
+ "max_active_tis_per_dag",
+ "max_active_tis_per_dagrun",
+ "max_retry_delay",
+ "multiple_outputs",
+ "on_failure_fail_dagrun",
+ "outlets",
+ "owner",
+ "params",
+ "partial_kwargs",
+ "pool",
+ "pool_slots",
+ "priority_weight",
+ "queue",
+ "render_template_as_native_obj",
+ "retries",
+ "retry_delay",
+ "retry_exponential_backoff",
+ "start_date",
+ "start_from_trigger",
+ "start_trigger_args",
+ "task_id",
+ "task_type",
+ "template_ext",
+ "template_fields",
+ "template_fields_renderers",
+ "trigger_rule",
+ "ui_color",
+ "ui_fgcolor",
+ "wait_for_downstream",
+ "wait_for_past_depends_before_skipping",
+ "weight_rule",
+ }
+)
+_DIFF_V1_REDACTED_SCHEMA_TASK_FIELDS = frozenset({"_arg_bindings"})
+# _get_category classifies task fields with these; every name must be a public task field or the
+# entry is unreachable, since a non-public field is aggregated under custom_fields before lookup.
+_DIFF_V1_TASK_ASSET_FIELDS = frozenset({"inlets", "outlets"})
+_DIFF_V1_TASK_PARAM_FIELDS = frozenset({"params"})
+_DIFF_V1_TASK_DEPENDENCY_FIELDS = frozenset({"downstream_task_ids"})
+_DIFF_V1_TASK_METADATA_FIELDS = frozenset(
+ {
+ "doc",
+ "doc_json",
+ "doc_md",
+ "doc_rst",
+ "doc_yaml",
+ "owner",
+ "ui_color",
+ "ui_fgcolor",
+ "_task_display_name",
+ "task_display_name",
+ }
+)
+_DIFF_V1_PUBLIC_PARTIAL_TASK_FIELDS = _DIFF_V1_PUBLIC_TASK_FIELDS | {"task_display_name"}
+# Classify every Dag schema field explicitly so new fields require a policy decision.
+_DIFF_V1_DAG_FIELD_CATEGORIES = {
+ "_concurrency": "schedule",
+ "_processor_dags_folder": "provenance",
+ "access_control": "authorization",
+ "allowed_run_types": "schedule",
+ "bundle_name": "provenance",
+ "catchup": "schedule",
+ "dag_dependencies": "dependency",
+ "dag_display_name": "metadata",
+ "dag_id": "metadata",
+ "dagrun_timeout": "schedule",
+ "deadline": "deadline",
+ "default_args": "param",
+ "description": "metadata",
+ "disable_bundle_versioning": "task",
+ "doc_md": "metadata",
+ "edge_info": "metadata",
+ "end_date": "schedule",
+ "fail_fast": "schedule",
+ "fileloc": "provenance",
+ "has_on_failure_callback": "callback",
+ "has_on_success_callback": "callback",
+ "is_paused_upon_creation": "schedule",
+ "max_active_runs": "schedule",
+ "max_active_tasks": "schedule",
+ "max_consecutive_failed_dag_runs": "schedule",
+ "owner_links": "metadata",
+ "params": "param",
+ "relative_fileloc": "provenance",
+ "render_template_as_native_obj": "task",
+ "rerun_with_latest_version": "task",
+ "start_date": "schedule",
+ "tags": "metadata",
+ "task_group": "task",
+ "tasks": "task",
+ "timetable": "schedule",
+ "timezone": "schedule",
+}
+_DIFF_V1_LEGACY_DAG_FIELD_CATEGORIES = {
+ "fail_stop": "schedule",
+ "on_failure_callback": "callback",
+ "on_success_callback": "callback",
+ "schedule": "schedule",
+ "schedule_interval": "schedule",
+}
+_RECURSIVE_MAPPING_PATHS = {
+ (),
+ ("dag",),
+ ("provenance",),
+ *_KEYED_COLLECTION_PATHS,
+}
+_DIFF_V1_TASK_GROUP_METADATA_FIELDS = frozenset(
+ {"group_display_name", "tooltip", "doc_md", "ui_color", "ui_fgcolor"}
+)
+_DIFF_V1_PUBLIC_TASK_GROUP_FIELDS = _DIFF_V1_TASK_GROUP_METADATA_FIELDS | {
+ "_group_id",
+ "prefix_group_id",
+ "children",
+ "upstream_group_ids",
+ "downstream_group_ids",
+ "upstream_task_ids",
+ "downstream_task_ids",
+ "expand_input",
+ "is_mapped",
+}
+
+
+def build_unavailable_dag_diff(
+ *,
+ base_data: dict[str, Any] | None,
+ target_data: dict[str, Any] | None,
+ reason: str,
+) -> dict[str, Any]:
+ """Report a known unavailable reason without comparing the stored payloads."""
+ return _mark_unavailable(
+ _build_diff_result(_get_schema_version(base_data), _get_schema_version(target_data)), reason
+ )
+
+
+def build_serialized_dag_diff(
+ *,
+ base_data: dict[str, Any] | None,
+ target_data: dict[str, Any] | None,
+ base_provenance: Mapping[str, Any] | None = None,
+ target_provenance: Mapping[str, Any] | None = None,
+ include_values: bool = False,
+ max_changes: int = DEFAULT_MAX_CHANGES,
+) -> dict[str, Any]:
+ """
+ Build a bounded, deterministic diff from two stored serialized Dag payloads.
+
+ Raw values, digests, and value-derived path components are returned only when
+ ``include_values`` is true. Callers must authorize disclosure of the entire
+ serialized payload, including access-control role names and permission mappings,
+ before enabling it.
+ """
+ validate_max_changes(max_changes)
+
+ base_schema_version = _get_schema_version(base_data)
+ target_schema_version = _get_schema_version(target_data)
+ result = _build_diff_result(base_schema_version, target_schema_version)
+
+ if base_data is None or target_data is None:
+ return _mark_unavailable(result, "serialized_dag_missing")
+
+ if base_schema_version is None or target_schema_version is None:
+ return _mark_unavailable(result, "serialized_dag_schema_version_missing")
+
+ unsupported_versions = [
+ version
+ for version in (base_schema_version, target_schema_version)
+ if version not in SUPPORTED_SERIALIZED_DAG_SCHEMA_VERSIONS
+ ]
+ if unsupported_versions:
+ return _mark_unavailable(
+ result, f"unsupported_serialized_dag_schema_version:{unsupported_versions[0]}"
+ )
+
+ try:
+ base_document = _canonicalize_payload_v1(base_data)
+ target_document = _canonicalize_payload_v1(target_data)
+ base_document["provenance"] = _canonicalize_value(dict(base_provenance or {}), path=("provenance",))
+ target_document["provenance"] = _canonicalize_value(
+ dict(target_provenance or {}), path=("provenance",)
+ )
+ except (AttributeError, KeyError, OverflowError, TypeError, ValueError) as error:
+ log.warning(
+ "Serialized Dag diff canonicalization failed",
+ error_type=type(error).__name__,
+ base_schema_version=base_schema_version,
+ target_schema_version=target_schema_version,
+ )
+ return _mark_unavailable(result, "serialized_dag_canonicalization_failed")
+
+ collector = _ChangeCollector(max_changes=max_changes, include_values=include_values)
+ try:
+ _collect_changes(base_document, target_document, path=(), collector=collector)
+ except _JsonEncodingError:
+ log.warning(
+ "Serialized Dag diff JSON encoding failed",
+ base_schema_version=base_schema_version,
+ target_schema_version=target_schema_version,
+ )
+ return _mark_unavailable(result, "serialized_dag_json_encoding_failed")
+
+ result["changes"] = collector.changes
+ result["truncated"] = collector.is_truncated
+ if include_values:
+ result["values"] = {"status": "available"}
+ return result
+
+
+class _ChangeCollector:
+ def __init__(self, *, max_changes: int, include_values: bool) -> None:
+ self.changes: list[dict[str, Any]] = []
+ self.count = 0
+ self.max_changes = max_changes
+ self.include_values = include_values
+
+ @property
+ def is_truncated(self) -> bool:
+ return self.count > self.max_changes
+
+ def add(
+ self,
+ *,
+ path: tuple[str, ...],
+ operation: Literal["added", "removed", "changed"],
+ before: Any,
+ after: Any,
+ ) -> None:
+ self.count += 1
+ if len(self.changes) >= self.max_changes:
+ return
+
+ public_path = _get_public_path(path)
+ category = _get_category(public_path)
+ change: dict[str, Any] = {
+ "path": _format_path(path if self.include_values else public_path),
+ "operation": operation,
+ "category": category,
+ "impact": _get_impact(category),
+ }
+ if self.include_values:
+ change["before_digest"] = None if before is _MISSING else _get_digest(before)
+ change["after_digest"] = None if after is _MISSING else _get_digest(after)
+ if before is not _MISSING:
+ change["before_value"] = before
+ if after is not _MISSING:
+ change["after_value"] = after
+ self.changes.append(change)
+
+
+_MISSING = object()
+
+
+def validate_max_changes(max_changes: int) -> None:
+ if not isinstance(max_changes, int) or isinstance(max_changes, bool) or max_changes < 1:
+ raise ValueError("max_changes must be a positive integer")
+ if max_changes > MAX_ALLOWED_CHANGES:
+ raise ValueError(f"max_changes must not exceed {MAX_ALLOWED_CHANGES}")
+
+
+def _get_schema_version(data: Mapping[str, Any] | None) -> int | None:
+ if not isinstance(data, Mapping):
+ return None
+ version = data.get("__version")
+ return version if isinstance(version, int) and not isinstance(version, bool) else None
+
+
+def _mark_unavailable(result: dict[str, Any], reason: str) -> dict[str, Any]:
+ result["mode"] = "unavailable"
+ result["unavailable_reason"] = reason
+ return result
+
+
+def _build_diff_result(base_schema_version: int | None, target_schema_version: int | None) -> dict[str, Any]:
+ return {
+ "diff_schema_version": DIFF_SCHEMA_VERSION,
+ "serialized_dag_schema_versions": {
+ "base": base_schema_version,
+ "target": target_schema_version,
+ },
+ "mode": "observed_state",
+ "changes": [],
+ "truncated": False,
+ # Always present so that reading result["values"]["status"] is safe for every caller
+ # of every entry point, whatever the outcome.
+ "values": {"status": "unavailable"},
+ }
+
+
+def _canonicalize_payload_v1(data: dict[str, Any]) -> dict[str, Any]:
+ payload = copy.deepcopy(data)
+ version = _get_schema_version(payload)
+ if version is None:
+ raise ValueError("missing or invalid __version")
+ if version == 1:
+ DagSerialization.conversion_v1_to_v2(payload)
+ DagSerialization.conversion_v2_to_v3(payload)
+ elif version == 2:
+ DagSerialization.conversion_v2_to_v3(payload)
+ if not isinstance(payload.get("dag"), Mapping):
+ raise ValueError("missing dag object")
+ dag_defaults = {
+ field: value
+ for field, value in DagSerialization.get_schema_defaults("dag").items()
+ # Dag callback flags are enabled by their presence, even when their value is false.
+ if field not in _DAG_CALLBACK_FIELDS
+ }
+ payload["dag"] = {**dag_defaults, **payload["dag"]}
+ for field in _DAG_CALLBACK_FIELDS & payload["dag"].keys():
+ payload["dag"][field] = True
+ if "params" in payload["dag"]:
+ payload["dag"]["params"] = _normalize_params(payload["dag"]["params"])
+ _apply_task_defaults(payload)
+ payload.pop("__version", None)
+ return _canonicalize_value(payload, path=())
+
+
+def _apply_task_defaults(payload: dict[str, Any]) -> None:
+ client_defaults = payload.pop("client_defaults", None)
+ if client_defaults is None:
+ client_defaults = {}
+ if not isinstance(client_defaults, Mapping):
+ raise ValueError("client_defaults is not an object")
+
+ # Fail loudly on a section this version cannot fold in: dropping it would silently
+ # compare two payloads as equal when the unhandled defaults actually differ.
+ if unknown_sections := client_defaults.keys() - {"tasks"}:
+ raise ValueError(f"unsupported client_defaults sections: {sorted(unknown_sections)}")
+
+ task_defaults = client_defaults.get("tasks", {})
+ if not isinstance(task_defaults, Mapping):
+ raise ValueError("client_defaults.tasks is not an object")
+
+ schema_defaults = DagSerialization.get_schema_defaults("operator")
+ partial_fields = (
+ SerializedBaseOperator.get_serialized_fields() - SerializedMappedOperator.get_serialized_fields()
+ )
+ # A mapped task resolves these through partial_kwargs, so their defaults have to be applied
+ # there and encoded like any other partial value rather than left at the outer level.
+ partial_schema_defaults = {
+ field: value for field, value in schema_defaults.items() if field in partial_fields
+ }
+ outer_schema_defaults = {
+ field: value for field, value in schema_defaults.items() if field not in partial_fields
+ }
+ # set_task_dag_references falls back to the Dag's dates, and the serializer elides a task date
+ # that already matches, so an absent task date means the Dag's date rather than a change.
+ inherited_dates = {field: payload["dag"].get(field) for field in ("start_date", "end_date")}
+ tasks = payload["dag"].get("tasks", [])
+ if not isinstance(tasks, list):
+ raise ValueError("dag.tasks is not a list")
+ for task in tasks:
+ if not isinstance(task, dict) or not isinstance(task.get("__var"), Mapping):
+ raise ValueError("task entry is not an object")
+ encoded_task = OperatorSerialization._apply_defaults_to_encoded_op(
+ dict(task["__var"]), dict(client_defaults)
+ )
+ upgraded_task = OperatorSerialization._upgrade_encoded_operator(encoded_task)
+ # Operator identity is selected before client defaults are applied during hydration.
+ upgraded_task["_is_mapped"] = bool(task["__var"].get("_is_mapped", False))
+ template_fields = upgraded_task.get("template_fields", [])
+ if upgraded_task.get("_is_mapped"):
+ task_data = {**outer_schema_defaults, **upgraded_task}
+ partial_kwargs = task_data.get("partial_kwargs", {})
+ if not isinstance(partial_kwargs, Mapping):
+ raise ValueError("partial_kwargs is not an object")
+ # populate_operator only folds client defaults into partial_kwargs when the payload
+ # carries the key, so an absent one leaves the top-level value as the effective value.
+ effective_partial_kwargs = (
+ {field: _encode_partial_field_value(field, value) for field, value in task_defaults.items()}
+ if "partial_kwargs" in task_data
+ else {}
+ )
+ for field, value in partial_kwargs.items():
+ if isinstance(value, Mapping) and "__type" in value and "__var" in value:
+ effective_partial_kwargs[field] = value
+ else:
+ effective_partial_kwargs[field] = _encode_partial_field_value(field, value)
+ # Match populate_operator: partial values take precedence over outer task defaults.
+ for field in partial_fields & task_data.keys():
+ value = task_data.pop(field)
+ # Outer fields are already encoded unless template handling bypasses deserialization.
+ if field in template_fields:
+ value = _encode_json_value(value)
+ effective_partial_kwargs.setdefault(field, value)
+ for field, value in partial_schema_defaults.items():
+ effective_partial_kwargs.setdefault(field, _encode_partial_field_value(field, value))
+ task_data["partial_kwargs"] = effective_partial_kwargs
+ _normalize_retry_backoff(effective_partial_kwargs)
+ else:
+ task_data = {**schema_defaults, **upgraded_task}
+ _normalize_retry_backoff(task_data)
+ for field in _OPERATOR_TIMEDELTA_FIELDS & task_data.keys():
+ value = task_data[field]
+ task_data[field] = (
+ _encode_json_value(value)
+ if field in template_fields and field in upgraded_task
+ else _encode_partial_field_value(field, value)
+ )
+ for field, dag_date in inherited_dates.items():
+ if task_data.get(field) is None:
+ task_data[field] = _encode_partial_field_value(field, dag_date)
+ elif field in template_fields:
+ task_data[field] = _encode_json_value(task_data[field])
+ else:
+ task_data[field] = _encode_partial_field_value(field, task_data[field])
+ if "params" in task_data and "params" not in template_fields:
+ task_data["params"] = _normalize_params(task_data["params"])
+ task["__var"] = task_data
+
+
+def _normalize_params(params: Any) -> Any:
+ """Normalize legacy Params without losing the order used by the trigger form."""
+ if isinstance(params, Mapping):
+ # 2.9.2 and earlier stored params as a JSON object instead of ordered pairs.
+ pairs: Any = params.items()
+ elif isinstance(params, list):
+ pairs = params
+ else:
+ return params
+
+ normalized: dict[str, Any] = {}
+ for pair in pairs:
+ if not isinstance(pair, (list, tuple)) or len(pair) != 2:
+ # Leave an unrecognised shape to the raw comparison rather than guessing at it.
+ return params
+ name, value = pair
+ if isinstance(value, Mapping) and "__class" in value:
+ value = {
+ "default": _normalize_param_attribute(value.get("default")),
+ "description": _normalize_param_attribute(value.get("description")),
+ "schema": _normalize_param_attribute(value.get("schema") or {}),
+ "source": _normalize_param_attribute(value.get("source")),
+ }
+ else:
+ value = {
+ "default": _encode_json_value(value),
+ "description": None,
+ "schema": _encode_json_value({}),
+ "source": None,
+ }
+ normalized[str(name)] = value
+ return [[name, value] for name, value in normalized.items()]
+
+
+def _normalize_param_attribute(value: Any) -> Any:
+ # Match _deserialize_param's legacy-encoding detection without hydrating user objects.
+ if isinstance(value, Mapping) and "__type" in value:
+ return value
+ if isinstance(value, list) and all(isinstance(item, Mapping) and "__type" in item for item in value):
+ return value
+ return _encode_json_value(value)
+
+
+def _encode_partial_field_value(field: str, value: Any) -> Any:
+ if field in _OPERATOR_TIMEDELTA_FIELDS and value is not None:
+ return {"__type": "timedelta", "__var": value}
+ if field.endswith("_date") and value is not None and not isinstance(value, str):
+ return {"__type": "datetime", "__var": value}
+ if field == "resources":
+ # Resources serialize as a raw mapping; a plain dictionary has a dict envelope instead.
+ return value
+ return _encode_json_value(value)
+
+
+def _encode_json_value(value: Any) -> Any:
+ """Encode plain JSON without invoking object serializers."""
+ if isinstance(value, Mapping):
+ return {"__type": "dict", "__var": {key: _encode_json_value(item) for key, item in value.items()}}
+ if isinstance(value, list):
+ return [_encode_json_value(item) for item in value]
+ return value
+
+
+def _normalize_retry_backoff(task_fields: dict[str, Any]) -> None:
+ if "retry_exponential_backoff" in task_fields:
+ value = task_fields["retry_exponential_backoff"]
+ task_fields["retry_exponential_backoff"] = 2.0 if value is True else float(value)
+
+
+def _canonicalize_value(value: Any, *, path: tuple[str, ...]) -> Any:
+ if isinstance(value, Mapping):
+ if path == ("dag", "deadline"):
+ if value.get("__type") == "deadline_alert":
+ value = value["__var"]
+ value = {"name": None, **value}
+ interval = value.get("interval")
+ if isinstance(interval, (int, float)) and not isinstance(interval, bool):
+ # Diff schema v1 uses the SDK's version-2 timedelta encoding for legacy seconds.
+ value["interval"] = {
+ "__classname__": "datetime.timedelta",
+ "__version__": 2,
+ "__data__": timedelta(seconds=interval).total_seconds(),
+ }
+ return {
+ canonical_key: _canonicalize_value(item, path=path + (canonical_key,))
+ for canonical_key, item in (
+ (_canonicalize_mapping_key(key), item)
+ for key, item in sorted(value.items(), key=lambda item: _canonicalize_mapping_key(item[0]))
+ )
+ }
+ if isinstance(value, (list, tuple)):
+ canonical_values = [_canonicalize_value(item, path=path) for item in value]
+ if path == ("dag", "tasks"):
+ return _canonicalize_keyed_list(canonical_values, _get_task_id, path)
+ if path == ("dag", "dag_dependencies"):
+ return _canonicalize_keyed_list(canonical_values, _get_dependency_key, path)
+ if path in _ORDER_INSENSITIVE_LIST_PATHS:
+ return _canonicalize_keyed_list(canonical_values, _get_string_key, path)
+ return canonical_values
+ return value
+
+
+def _canonicalize_mapping_key(key: Any) -> str:
+ if isinstance(key, str) and isinstance(key, Enum):
+ return key.value
+ return str(key)
+
+
+def _canonicalize_keyed_list(
+ values: list[Any], key_getter: Callable[[Any], str], path: tuple[str, ...]
+) -> dict[str, Any]:
+ keyed_values: dict[str, Any] = {}
+ for value in values:
+ key = key_getter(value)
+ if key in keyed_values:
+ if path in _ORDER_INSENSITIVE_LIST_PATHS or (
+ path == ("dag", "dag_dependencies") and keyed_values[key] == value
+ ):
+ continue
+ raise ValueError(f"duplicate key {key!r} in /{'/'.join(path)}")
+ keyed_values[key] = value
+ return {key: keyed_values[key] for key in sorted(keyed_values)}
+
+
+def _get_task_id(task: Any) -> str:
+ if not isinstance(task, Mapping):
+ raise ValueError("task entry is not an object")
+ task_data = task.get("__var", task)
+ task_id = task_data.get("task_id") if isinstance(task_data, Mapping) else None
+ if not isinstance(task_id, str):
+ raise ValueError("task entry has no task_id")
+ return task_id
+
+
+def _get_string_key(value: Any) -> str:
+ if not isinstance(value, str):
+ raise ValueError("collection entry is not a string")
+ return value
+
+
+def _get_dependency_key(dependency: Any) -> str:
+ if not isinstance(dependency, Mapping):
+ raise ValueError("dependency entry is not an object")
+ components = (
+ dependency.get("dependency_type"),
+ dependency.get("dependency_id"),
+ dependency.get("source"),
+ dependency.get("target"),
+ dependency.get("label"),
+ )
+ return json.dumps(components, ensure_ascii=False, separators=(",", ":"))
+
+
+def _collect_changes(
+ before: Any,
+ after: Any,
+ *,
+ path: tuple[str, ...],
+ collector: _ChangeCollector,
+) -> None:
+ if before is _MISSING and after is _MISSING:
+ return
+ if isinstance(before, Mapping) and isinstance(after, Mapping):
+ # The walk shape never depends on ``include_values``: authorization decides what each
+ # change carries, not which changes exist, so ``max_changes`` means one thing and a
+ # caller allowed values never receives a less complete change set than a redacted one.
+ if _is_task_mapping_path(path):
+ _collect_task_changes(before, after, path=path, collector=collector)
+ return
+ if _is_task_group_mapping_path(path):
+ _collect_public_field_changes(
+ before,
+ after,
+ path=path,
+ public_fields=_DIFF_V1_PUBLIC_TASK_GROUP_FIELDS,
+ collector=collector,
+ )
+ return
+ if not _should_recurse_mapping(path):
+ if not _is_json_equal(before, after):
+ collector.add(path=path, operation="changed", before=before, after=after)
+ return
+ keys = sorted({str(key) for key in before} | {str(key) for key in after})
+ for key in keys:
+ before_value = before.get(key, _MISSING)
+ after_value = after.get(key, _MISSING)
+ _collect_changes(before_value, after_value, path=path + (key,), collector=collector)
+ if collector.is_truncated:
+ return
+ return
+
+ if isinstance(before, list) and isinstance(after, list):
+ if (
+ _is_task_group_child_path(path)
+ and len(before) == len(after) == 2
+ and before[0] == after[0] == "taskgroup"
+ and isinstance(before[1], Mapping)
+ and isinstance(after[1], Mapping)
+ ):
+ _collect_changes(before[1], after[1], path=path + ("1",), collector=collector)
+ return
+ if not _is_json_equal(before, after):
+ collector.add(path=path, operation="changed", before=before, after=after)
+ return
+
+ if before is _MISSING:
+ collector.add(path=path, operation="added", before=_MISSING, after=after)
+ elif after is _MISSING:
+ collector.add(path=path, operation="removed", before=before, after=_MISSING)
+ elif not _is_json_equal(before, after):
+ collector.add(path=path, operation="changed", before=before, after=after)
+
+
+def _collect_task_changes(
+ before: Mapping[str, Any],
+ after: Mapping[str, Any],
+ *,
+ path: tuple[str, ...],
+ collector: _ChangeCollector,
+) -> None:
+ if len(path) == 3:
+ _collect_changes(
+ before.get("__type", _MISSING),
+ after.get("__type", _MISSING),
+ path=path + ("__type",),
+ collector=collector,
+ )
+ if collector.is_truncated:
+ return
+ before, after = before["__var"], after["__var"]
+ public_fields = _DIFF_V1_PUBLIC_TASK_FIELDS - {"__type"}
+ else:
+ public_fields = _DIFF_V1_PUBLIC_PARTIAL_TASK_FIELDS
+ _collect_public_field_changes(before, after, path=path, public_fields=public_fields, collector=collector)
+
+
+def _collect_public_field_changes(
+ before: Mapping[str, Any],
+ after: Mapping[str, Any],
+ *,
+ path: tuple[str, ...],
+ public_fields: frozenset[str] | set[str],
+ collector: _ChangeCollector,
+) -> None:
+ """Walk allowlisted fields one by one and report everything else as a single change."""
+ keys = {str(key) for key in before} | {str(key) for key in after}
+ for key in sorted(keys & public_fields):
+ _collect_changes(
+ before.get(key, _MISSING),
+ after.get(key, _MISSING),
+ path=path + (key,),
+ collector=collector,
+ )
+ if collector.is_truncated:
+ return
+
+ before_custom_fields = {key: before[key] for key in before if key not in public_fields}
+ after_custom_fields = {key: after[key] for key in after if key not in public_fields}
+ if not _is_json_equal(before_custom_fields, after_custom_fields):
+ collector.add(
+ path=path + (_CUSTOM_TASK_FIELDS_PATH_COMPONENT,),
+ operation="changed",
+ before=before_custom_fields,
+ after=after_custom_fields,
+ )
+
+
+def _is_task_mapping_path(path: tuple[str, ...]) -> bool:
+ return path[:2] == ("dag", "tasks") and (
+ len(path) == 3 or (len(path) == 4 and path[3] == "partial_kwargs")
+ )
+
+
+def _is_task_group_mapping_path(path: tuple[str, ...]) -> bool:
+ index = _get_task_group_field_index(path)
+ return index is not None and len(path) == index
+
+
+def _should_recurse_mapping(path: tuple[str, ...]) -> bool:
+ if path in _RECURSIVE_MAPPING_PATHS:
+ return True
+ index = _get_task_group_field_index(path)
+ return index is not None and path[index:] == ("children",)
+
+
+def _get_task_group_field_index(path: tuple[str, ...]) -> int | None:
+ if path[:2] != ("dag", "task_group"):
+ return None
+ index = 2
+ while len(path) >= index + 3 and path[index] == "children" and path[index + 2] == "1":
+ index += 3
+ return index
+
+
+def _is_task_group_child_path(path: tuple[str, ...]) -> bool:
+ index = _get_task_group_field_index(path)
+ return index is not None and len(path) == index + 2 and path[index] == "children"
+
+
+def _get_public_path(path: tuple[str, ...]) -> tuple[str, ...]:
+ if path[:2] == ("dag", "task_group"):
+ public_path = list(path)
+ index = 2
+ while len(path) >= index + 2 and path[index] == "children":
+ public_path[index + 1] = "*"
+ if len(path) < index + 3 or path[index + 2] != "1":
+ return tuple(public_path)
+ index += 3
+ if len(path) > index and path[index] not in _DIFF_V1_PUBLIC_TASK_GROUP_FIELDS:
+ public_path[index] = "custom_fields"
+ return tuple(public_path)
+ if len(path) >= 3 and path[:2] in _KEYED_COLLECTION_PATHS:
+ path = (*path[:2], "*", *path[3:])
+ if len(path) >= 4 and path[:2] == ("dag", "tasks"):
+ field_index = 4 if path[3] == "partial_kwargs" and len(path) >= 5 else 3
+ public_fields = (
+ _DIFF_V1_PUBLIC_PARTIAL_TASK_FIELDS if field_index == 4 else _DIFF_V1_PUBLIC_TASK_FIELDS
+ )
+ if path[field_index] not in public_fields:
+ return (*path[:field_index], _CUSTOM_TASK_FIELDS_PATH_COMPONENT)
+ return path
+
+
+def _format_path(path: tuple[str, ...]) -> str:
+ return "/" + "/".join(component.replace("~", "~0").replace("/", "~1") for component in path)
+
+
+class _JsonEncodingError(TypeError):
+ """Distinguish unsupported JSON values from comparison implementation errors."""
+
+
+def _serialize_canonical_json(value: Any) -> str:
+ try:
+ return json.dumps(value, ensure_ascii=False, separators=(",", ":"), sort_keys=True)
+ except TypeError as error:
+ raise _JsonEncodingError from error
+
+
+def _is_json_equal(before: Any, after: Any) -> bool:
+ # Compare canonical JSON, not Python ==, so a JSON type change (1 vs 1.0, True vs 1,
+ # False vs 0) registers as a change instead of collapsing under Python equality.
+ return _serialize_canonical_json(before) == _serialize_canonical_json(after)
+
+
+def _get_digest(value: Any) -> str:
+ return f"sha256:{hashlib.sha256(_serialize_canonical_json(value).encode()).hexdigest()}"
Review Comment:
SHA-256 here fingerprints canonical serialized diff content; this code is not used for password storage or verification. CodeQL traces a value from `Connection.to_dict()["password"]`, but hashing content that may contain a password does not make this a password-protection mechanism.
`_ChangeCollector.add()` only produces digests when `include_values=True`, and each digest is returned alongside its corresponding raw value. `DagVersion.get_diff()` defaults to `values_status="unavailable"`; callers must authorize disclosure of the full serialized payload before enabling values. Redacted responses omit both values and digests, which existing regression tests verify.
The digest therefore reveals no additional value to the recipient beyond the raw content already authorized for disclosure. A password KDF would not improve protection here; SHA-256 serves deterministic content fingerprinting. This is a false positive for `py/weak-sensitive-data-hashing`.
---
Drafted-by: Codex (GPT-6) (no human review before posting)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add serialized Dag version diff engine [airflow]
Posted by "github-advanced-security[bot] (via GitHub)" <gi...@apache.org>.
github-advanced-security[bot] commented on code in PR #69864:
URL: https://github.com/apache/airflow/pull/69864#discussion_r4005422661
##########
airflow-core/src/airflow/serialization/dag_version_diff.py:
##########
@@ -0,0 +1,896 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""Observed-state diffs for serialized Dag payloads."""
+
+from __future__ import annotations
+
+import copy
+import hashlib
+import json
+from collections.abc import Callable, Mapping
+from datetime import timedelta
+from enum import Enum
+from typing import Any, Literal
+
+import structlog
+
+from airflow.serialization.definitions.baseoperator import SerializedBaseOperator
+from airflow.serialization.definitions.mappedoperator import SerializedMappedOperator
+from airflow.serialization.serialized_objects import (
+ _DAG_CALLBACK_FIELDS,
+ _OPERATOR_TIMEDELTA_FIELDS,
+ DagSerialization,
+ OperatorSerialization,
+)
+
+log = structlog.get_logger(__name__)
+
+DIFF_SCHEMA_VERSION = 1
+DEFAULT_MAX_CHANGES = 500
+MAX_ALLOWED_CHANGES = 5000
+SUPPORTED_SERIALIZED_DAG_SCHEMA_VERSIONS = frozenset((1, 2, 3))
+
+_ORDER_INSENSITIVE_LIST_PATHS = {
+ ("dag", "tags"),
+ ("dag", "allowed_run_types"),
+}
+_KEYED_COLLECTION_PATHS = {
+ ("dag", "tasks"),
+ ("dag", "dag_dependencies"),
+ *_ORDER_INSENSITIVE_LIST_PATHS,
+}
+_CUSTOM_TASK_FIELDS_PATH_COMPONENT = "custom_fields"
+# This allowlist is part of diff schema v1. Serializer schema changes must not
+# silently change the paths visible to callers of the diff API.
+_DIFF_V1_PUBLIC_TASK_FIELDS = frozenset(
+ {
+ "__type",
+ "_disallow_kwargs_override",
+ "_expand_input_attr",
+ "_is_mapped",
+ "_is_sensor",
+ "_logger_name",
+ "_needs_expansion",
+ "_operator_extra_links",
+ "_task_display_name",
+ "_task_module",
+ "allow_nested_operators",
+ "depends_on_past",
+ "do_xcom_push",
+ "doc",
+ "doc_json",
+ "doc_md",
+ "doc_rst",
+ "doc_yaml",
+ "downstream_task_ids",
+ "email_on_failure",
+ "email_on_retry",
+ "end_date",
+ "execution_timeout",
+ "executor",
+ "executor_config",
+ "has_on_execute_callback",
+ "has_on_failure_callback",
+ "has_on_retry_callback",
+ "has_on_skipped_callback",
+ "has_on_success_callback",
+ "ignore_first_depends_on_past",
+ "inlets",
+ "is_setup",
+ "is_teardown",
+ "map_index_template",
+ "max_active_tis_per_dag",
+ "max_active_tis_per_dagrun",
+ "max_retry_delay",
+ "multiple_outputs",
+ "on_failure_fail_dagrun",
+ "outlets",
+ "owner",
+ "params",
+ "partial_kwargs",
+ "pool",
+ "pool_slots",
+ "priority_weight",
+ "queue",
+ "render_template_as_native_obj",
+ "retries",
+ "retry_delay",
+ "retry_exponential_backoff",
+ "start_date",
+ "start_from_trigger",
+ "start_trigger_args",
+ "task_id",
+ "task_type",
+ "template_ext",
+ "template_fields",
+ "template_fields_renderers",
+ "trigger_rule",
+ "ui_color",
+ "ui_fgcolor",
+ "wait_for_downstream",
+ "wait_for_past_depends_before_skipping",
+ "weight_rule",
+ }
+)
+_DIFF_V1_REDACTED_SCHEMA_TASK_FIELDS = frozenset({"_arg_bindings"})
+# _get_category classifies task fields with these; every name must be a public task field or the
+# entry is unreachable, since a non-public field is aggregated under custom_fields before lookup.
+_DIFF_V1_TASK_ASSET_FIELDS = frozenset({"inlets", "outlets"})
+_DIFF_V1_TASK_PARAM_FIELDS = frozenset({"params"})
+_DIFF_V1_TASK_DEPENDENCY_FIELDS = frozenset({"downstream_task_ids"})
+_DIFF_V1_TASK_METADATA_FIELDS = frozenset(
+ {
+ "doc",
+ "doc_json",
+ "doc_md",
+ "doc_rst",
+ "doc_yaml",
+ "owner",
+ "ui_color",
+ "ui_fgcolor",
+ "_task_display_name",
+ "task_display_name",
+ }
+)
+_DIFF_V1_PUBLIC_PARTIAL_TASK_FIELDS = _DIFF_V1_PUBLIC_TASK_FIELDS | {"task_display_name"}
+# Classify every Dag schema field explicitly so new fields require a policy decision.
+_DIFF_V1_DAG_FIELD_CATEGORIES = {
+ "_concurrency": "schedule",
+ "_processor_dags_folder": "provenance",
+ "access_control": "authorization",
+ "allowed_run_types": "schedule",
+ "bundle_name": "provenance",
+ "catchup": "schedule",
+ "dag_dependencies": "dependency",
+ "dag_display_name": "metadata",
+ "dag_id": "metadata",
+ "dagrun_timeout": "schedule",
+ "deadline": "deadline",
+ "default_args": "param",
+ "description": "metadata",
+ "disable_bundle_versioning": "task",
+ "doc_md": "metadata",
+ "edge_info": "metadata",
+ "end_date": "schedule",
+ "fail_fast": "schedule",
+ "fileloc": "provenance",
+ "has_on_failure_callback": "callback",
+ "has_on_success_callback": "callback",
+ "is_paused_upon_creation": "schedule",
+ "max_active_runs": "schedule",
+ "max_active_tasks": "schedule",
+ "max_consecutive_failed_dag_runs": "schedule",
+ "owner_links": "metadata",
+ "params": "param",
+ "relative_fileloc": "provenance",
+ "render_template_as_native_obj": "task",
+ "rerun_with_latest_version": "task",
+ "start_date": "schedule",
+ "tags": "metadata",
+ "task_group": "task",
+ "tasks": "task",
+ "timetable": "schedule",
+ "timezone": "schedule",
+}
+_DIFF_V1_LEGACY_DAG_FIELD_CATEGORIES = {
+ "fail_stop": "schedule",
+ "on_failure_callback": "callback",
+ "on_success_callback": "callback",
+ "schedule": "schedule",
+ "schedule_interval": "schedule",
+}
+_RECURSIVE_MAPPING_PATHS = {
+ (),
+ ("dag",),
+ ("provenance",),
+ *_KEYED_COLLECTION_PATHS,
+}
+_DIFF_V1_TASK_GROUP_METADATA_FIELDS = frozenset(
+ {"group_display_name", "tooltip", "doc_md", "ui_color", "ui_fgcolor"}
+)
+_DIFF_V1_PUBLIC_TASK_GROUP_FIELDS = _DIFF_V1_TASK_GROUP_METADATA_FIELDS | {
+ "_group_id",
+ "prefix_group_id",
+ "children",
+ "upstream_group_ids",
+ "downstream_group_ids",
+ "upstream_task_ids",
+ "downstream_task_ids",
+ "expand_input",
+ "is_mapped",
+}
+
+
+def build_unavailable_dag_diff(
+ *,
+ base_data: dict[str, Any] | None,
+ target_data: dict[str, Any] | None,
+ reason: str,
+) -> dict[str, Any]:
+ """Report a known unavailable reason without comparing the stored payloads."""
+ return _mark_unavailable(
+ _build_diff_result(_get_schema_version(base_data), _get_schema_version(target_data)), reason
+ )
+
+
+def build_serialized_dag_diff(
+ *,
+ base_data: dict[str, Any] | None,
+ target_data: dict[str, Any] | None,
+ base_provenance: Mapping[str, Any] | None = None,
+ target_provenance: Mapping[str, Any] | None = None,
+ include_values: bool = False,
+ max_changes: int = DEFAULT_MAX_CHANGES,
+) -> dict[str, Any]:
+ """
+ Build a bounded, deterministic diff from two stored serialized Dag payloads.
+
+ Raw values, digests, and value-derived path components are returned only when
+ ``include_values`` is true. Callers must authorize disclosure of the entire
+ serialized payload, including access-control role names and permission mappings,
+ before enabling it.
+ """
+ validate_max_changes(max_changes)
+
+ base_schema_version = _get_schema_version(base_data)
+ target_schema_version = _get_schema_version(target_data)
+ result = _build_diff_result(base_schema_version, target_schema_version)
+
+ if base_data is None or target_data is None:
+ return _mark_unavailable(result, "serialized_dag_missing")
+
+ if base_schema_version is None or target_schema_version is None:
+ return _mark_unavailable(result, "serialized_dag_schema_version_missing")
+
+ unsupported_versions = [
+ version
+ for version in (base_schema_version, target_schema_version)
+ if version not in SUPPORTED_SERIALIZED_DAG_SCHEMA_VERSIONS
+ ]
+ if unsupported_versions:
+ return _mark_unavailable(
+ result, f"unsupported_serialized_dag_schema_version:{unsupported_versions[0]}"
+ )
+
+ try:
+ base_document = _canonicalize_payload_v1(base_data)
+ target_document = _canonicalize_payload_v1(target_data)
+ base_document["provenance"] = _canonicalize_value(dict(base_provenance or {}), path=("provenance",))
+ target_document["provenance"] = _canonicalize_value(
+ dict(target_provenance or {}), path=("provenance",)
+ )
+ except (AttributeError, KeyError, OverflowError, TypeError, ValueError) as error:
+ log.warning(
+ "Serialized Dag diff canonicalization failed",
+ error_type=type(error).__name__,
+ base_schema_version=base_schema_version,
+ target_schema_version=target_schema_version,
+ )
+ return _mark_unavailable(result, "serialized_dag_canonicalization_failed")
+
+ collector = _ChangeCollector(max_changes=max_changes, include_values=include_values)
+ try:
+ _collect_changes(base_document, target_document, path=(), collector=collector)
+ except _JsonEncodingError:
+ log.warning(
+ "Serialized Dag diff JSON encoding failed",
+ base_schema_version=base_schema_version,
+ target_schema_version=target_schema_version,
+ )
+ return _mark_unavailable(result, "serialized_dag_json_encoding_failed")
+
+ result["changes"] = collector.changes
+ result["truncated"] = collector.is_truncated
+ if include_values:
+ result["values"] = {"status": "available"}
+ return result
+
+
+class _ChangeCollector:
+ def __init__(self, *, max_changes: int, include_values: bool) -> None:
+ self.changes: list[dict[str, Any]] = []
+ self.count = 0
+ self.max_changes = max_changes
+ self.include_values = include_values
+
+ @property
+ def is_truncated(self) -> bool:
+ return self.count > self.max_changes
+
+ def add(
+ self,
+ *,
+ path: tuple[str, ...],
+ operation: Literal["added", "removed", "changed"],
+ before: Any,
+ after: Any,
+ ) -> None:
+ self.count += 1
+ if len(self.changes) >= self.max_changes:
+ return
+
+ public_path = _get_public_path(path)
+ category = _get_category(public_path)
+ change: dict[str, Any] = {
+ "path": _format_path(path if self.include_values else public_path),
+ "operation": operation,
+ "category": category,
+ "impact": _get_impact(category),
+ }
+ if self.include_values:
+ change["before_digest"] = None if before is _MISSING else _get_digest(before)
+ change["after_digest"] = None if after is _MISSING else _get_digest(after)
+ if before is not _MISSING:
+ change["before_value"] = before
+ if after is not _MISSING:
+ change["after_value"] = after
+ self.changes.append(change)
+
+
+_MISSING = object()
+
+
+def validate_max_changes(max_changes: int) -> None:
+ if not isinstance(max_changes, int) or isinstance(max_changes, bool) or max_changes < 1:
+ raise ValueError("max_changes must be a positive integer")
+ if max_changes > MAX_ALLOWED_CHANGES:
+ raise ValueError(f"max_changes must not exceed {MAX_ALLOWED_CHANGES}")
+
+
+def _get_schema_version(data: Mapping[str, Any] | None) -> int | None:
+ if not isinstance(data, Mapping):
+ return None
+ version = data.get("__version")
+ return version if isinstance(version, int) and not isinstance(version, bool) else None
+
+
+def _mark_unavailable(result: dict[str, Any], reason: str) -> dict[str, Any]:
+ result["mode"] = "unavailable"
+ result["unavailable_reason"] = reason
+ return result
+
+
+def _build_diff_result(base_schema_version: int | None, target_schema_version: int | None) -> dict[str, Any]:
+ return {
+ "diff_schema_version": DIFF_SCHEMA_VERSION,
+ "serialized_dag_schema_versions": {
+ "base": base_schema_version,
+ "target": target_schema_version,
+ },
+ "mode": "observed_state",
+ "changes": [],
+ "truncated": False,
+ # Always present so that reading result["values"]["status"] is safe for every caller
+ # of every entry point, whatever the outcome.
+ "values": {"status": "unavailable"},
+ }
+
+
+def _canonicalize_payload_v1(data: dict[str, Any]) -> dict[str, Any]:
+ payload = copy.deepcopy(data)
+ version = _get_schema_version(payload)
+ if version is None:
+ raise ValueError("missing or invalid __version")
+ if version == 1:
+ DagSerialization.conversion_v1_to_v2(payload)
+ DagSerialization.conversion_v2_to_v3(payload)
+ elif version == 2:
+ DagSerialization.conversion_v2_to_v3(payload)
+ if not isinstance(payload.get("dag"), Mapping):
+ raise ValueError("missing dag object")
+ dag_defaults = {
+ field: value
+ for field, value in DagSerialization.get_schema_defaults("dag").items()
+ # Dag callback flags are enabled by their presence, even when their value is false.
+ if field not in _DAG_CALLBACK_FIELDS
+ }
+ payload["dag"] = {**dag_defaults, **payload["dag"]}
+ for field in _DAG_CALLBACK_FIELDS & payload["dag"].keys():
+ payload["dag"][field] = True
+ if "params" in payload["dag"]:
+ payload["dag"]["params"] = _normalize_params(payload["dag"]["params"])
+ _apply_task_defaults(payload)
+ payload.pop("__version", None)
+ return _canonicalize_value(payload, path=())
+
+
+def _apply_task_defaults(payload: dict[str, Any]) -> None:
+ client_defaults = payload.pop("client_defaults", None)
+ if client_defaults is None:
+ client_defaults = {}
+ if not isinstance(client_defaults, Mapping):
+ raise ValueError("client_defaults is not an object")
+
+ # Fail loudly on a section this version cannot fold in: dropping it would silently
+ # compare two payloads as equal when the unhandled defaults actually differ.
+ if unknown_sections := client_defaults.keys() - {"tasks"}:
+ raise ValueError(f"unsupported client_defaults sections: {sorted(unknown_sections)}")
+
+ task_defaults = client_defaults.get("tasks", {})
+ if not isinstance(task_defaults, Mapping):
+ raise ValueError("client_defaults.tasks is not an object")
+
+ schema_defaults = DagSerialization.get_schema_defaults("operator")
+ partial_fields = (
+ SerializedBaseOperator.get_serialized_fields() - SerializedMappedOperator.get_serialized_fields()
+ )
+ # A mapped task resolves these through partial_kwargs, so their defaults have to be applied
+ # there and encoded like any other partial value rather than left at the outer level.
+ partial_schema_defaults = {
+ field: value for field, value in schema_defaults.items() if field in partial_fields
+ }
+ outer_schema_defaults = {
+ field: value for field, value in schema_defaults.items() if field not in partial_fields
+ }
+ # set_task_dag_references falls back to the Dag's dates, and the serializer elides a task date
+ # that already matches, so an absent task date means the Dag's date rather than a change.
+ inherited_dates = {field: payload["dag"].get(field) for field in ("start_date", "end_date")}
+ tasks = payload["dag"].get("tasks", [])
+ if not isinstance(tasks, list):
+ raise ValueError("dag.tasks is not a list")
+ for task in tasks:
+ if not isinstance(task, dict) or not isinstance(task.get("__var"), Mapping):
+ raise ValueError("task entry is not an object")
+ encoded_task = OperatorSerialization._apply_defaults_to_encoded_op(
+ dict(task["__var"]), dict(client_defaults)
+ )
+ upgraded_task = OperatorSerialization._upgrade_encoded_operator(encoded_task)
+ # Operator identity is selected before client defaults are applied during hydration.
+ upgraded_task["_is_mapped"] = bool(task["__var"].get("_is_mapped", False))
+ template_fields = upgraded_task.get("template_fields", [])
+ if upgraded_task.get("_is_mapped"):
+ task_data = {**outer_schema_defaults, **upgraded_task}
+ partial_kwargs = task_data.get("partial_kwargs", {})
+ if not isinstance(partial_kwargs, Mapping):
+ raise ValueError("partial_kwargs is not an object")
+ # populate_operator only folds client defaults into partial_kwargs when the payload
+ # carries the key, so an absent one leaves the top-level value as the effective value.
+ effective_partial_kwargs = (
+ {field: _encode_partial_field_value(field, value) for field, value in task_defaults.items()}
+ if "partial_kwargs" in task_data
+ else {}
+ )
+ for field, value in partial_kwargs.items():
+ if isinstance(value, Mapping) and "__type" in value and "__var" in value:
+ effective_partial_kwargs[field] = value
+ else:
+ effective_partial_kwargs[field] = _encode_partial_field_value(field, value)
+ # Match populate_operator: partial values take precedence over outer task defaults.
+ for field in partial_fields & task_data.keys():
+ value = task_data.pop(field)
+ # Outer fields are already encoded unless template handling bypasses deserialization.
+ if field in template_fields:
+ value = _encode_json_value(value)
+ effective_partial_kwargs.setdefault(field, value)
+ for field, value in partial_schema_defaults.items():
+ effective_partial_kwargs.setdefault(field, _encode_partial_field_value(field, value))
+ task_data["partial_kwargs"] = effective_partial_kwargs
+ _normalize_retry_backoff(effective_partial_kwargs)
+ else:
+ task_data = {**schema_defaults, **upgraded_task}
+ _normalize_retry_backoff(task_data)
+ for field in _OPERATOR_TIMEDELTA_FIELDS & task_data.keys():
+ value = task_data[field]
+ task_data[field] = (
+ _encode_json_value(value)
+ if field in template_fields and field in upgraded_task
+ else _encode_partial_field_value(field, value)
+ )
+ for field, dag_date in inherited_dates.items():
+ if task_data.get(field) is None:
+ task_data[field] = _encode_partial_field_value(field, dag_date)
+ elif field in template_fields:
+ task_data[field] = _encode_json_value(task_data[field])
+ else:
+ task_data[field] = _encode_partial_field_value(field, task_data[field])
+ if "params" in task_data and "params" not in template_fields:
+ task_data["params"] = _normalize_params(task_data["params"])
+ task["__var"] = task_data
+
+
+def _normalize_params(params: Any) -> Any:
+ """Normalize legacy Params without losing the order used by the trigger form."""
+ if isinstance(params, Mapping):
+ # 2.9.2 and earlier stored params as a JSON object instead of ordered pairs.
+ pairs: Any = params.items()
+ elif isinstance(params, list):
+ pairs = params
+ else:
+ return params
+
+ normalized: dict[str, Any] = {}
+ for pair in pairs:
+ if not isinstance(pair, (list, tuple)) or len(pair) != 2:
+ # Leave an unrecognised shape to the raw comparison rather than guessing at it.
+ return params
+ name, value = pair
+ if isinstance(value, Mapping) and "__class" in value:
+ value = {
+ "default": _normalize_param_attribute(value.get("default")),
+ "description": _normalize_param_attribute(value.get("description")),
+ "schema": _normalize_param_attribute(value.get("schema") or {}),
+ "source": _normalize_param_attribute(value.get("source")),
+ }
+ else:
+ value = {
+ "default": _encode_json_value(value),
+ "description": None,
+ "schema": _encode_json_value({}),
+ "source": None,
+ }
+ normalized[str(name)] = value
+ return [[name, value] for name, value in normalized.items()]
+
+
+def _normalize_param_attribute(value: Any) -> Any:
+ # Match _deserialize_param's legacy-encoding detection without hydrating user objects.
+ if isinstance(value, Mapping) and "__type" in value:
+ return value
+ if isinstance(value, list) and all(isinstance(item, Mapping) and "__type" in item for item in value):
+ return value
+ return _encode_json_value(value)
+
+
+def _encode_partial_field_value(field: str, value: Any) -> Any:
+ if field in _OPERATOR_TIMEDELTA_FIELDS and value is not None:
+ return {"__type": "timedelta", "__var": value}
+ if field.endswith("_date") and value is not None and not isinstance(value, str):
+ return {"__type": "datetime", "__var": value}
+ if field == "resources":
+ # Resources serialize as a raw mapping; a plain dictionary has a dict envelope instead.
+ return value
+ return _encode_json_value(value)
+
+
+def _encode_json_value(value: Any) -> Any:
+ """Encode plain JSON without invoking object serializers."""
+ if isinstance(value, Mapping):
+ return {"__type": "dict", "__var": {key: _encode_json_value(item) for key, item in value.items()}}
+ if isinstance(value, list):
+ return [_encode_json_value(item) for item in value]
+ return value
+
+
+def _normalize_retry_backoff(task_fields: dict[str, Any]) -> None:
+ if "retry_exponential_backoff" in task_fields:
+ value = task_fields["retry_exponential_backoff"]
+ task_fields["retry_exponential_backoff"] = 2.0 if value is True else float(value)
+
+
+def _canonicalize_value(value: Any, *, path: tuple[str, ...]) -> Any:
+ if isinstance(value, Mapping):
+ if path == ("dag", "deadline"):
+ if value.get("__type") == "deadline_alert":
+ value = value["__var"]
+ value = {"name": None, **value}
+ interval = value.get("interval")
+ if isinstance(interval, (int, float)) and not isinstance(interval, bool):
+ # Diff schema v1 uses the SDK's version-2 timedelta encoding for legacy seconds.
+ value["interval"] = {
+ "__classname__": "datetime.timedelta",
+ "__version__": 2,
+ "__data__": timedelta(seconds=interval).total_seconds(),
+ }
+ return {
+ canonical_key: _canonicalize_value(item, path=path + (canonical_key,))
+ for canonical_key, item in (
+ (_canonicalize_mapping_key(key), item)
+ for key, item in sorted(value.items(), key=lambda item: _canonicalize_mapping_key(item[0]))
+ )
+ }
+ if isinstance(value, (list, tuple)):
+ canonical_values = [_canonicalize_value(item, path=path) for item in value]
+ if path == ("dag", "tasks"):
+ return _canonicalize_keyed_list(canonical_values, _get_task_id, path)
+ if path == ("dag", "dag_dependencies"):
+ return _canonicalize_keyed_list(canonical_values, _get_dependency_key, path)
+ if path in _ORDER_INSENSITIVE_LIST_PATHS:
+ return _canonicalize_keyed_list(canonical_values, _get_string_key, path)
+ return canonical_values
+ return value
+
+
+def _canonicalize_mapping_key(key: Any) -> str:
+ if isinstance(key, str) and isinstance(key, Enum):
+ return key.value
+ return str(key)
+
+
+def _canonicalize_keyed_list(
+ values: list[Any], key_getter: Callable[[Any], str], path: tuple[str, ...]
+) -> dict[str, Any]:
+ keyed_values: dict[str, Any] = {}
+ for value in values:
+ key = key_getter(value)
+ if key in keyed_values:
+ if path in _ORDER_INSENSITIVE_LIST_PATHS or (
+ path == ("dag", "dag_dependencies") and keyed_values[key] == value
+ ):
+ continue
+ raise ValueError(f"duplicate key {key!r} in /{'/'.join(path)}")
+ keyed_values[key] = value
+ return {key: keyed_values[key] for key in sorted(keyed_values)}
+
+
+def _get_task_id(task: Any) -> str:
+ if not isinstance(task, Mapping):
+ raise ValueError("task entry is not an object")
+ task_data = task.get("__var", task)
+ task_id = task_data.get("task_id") if isinstance(task_data, Mapping) else None
+ if not isinstance(task_id, str):
+ raise ValueError("task entry has no task_id")
+ return task_id
+
+
+def _get_string_key(value: Any) -> str:
+ if not isinstance(value, str):
+ raise ValueError("collection entry is not a string")
+ return value
+
+
+def _get_dependency_key(dependency: Any) -> str:
+ if not isinstance(dependency, Mapping):
+ raise ValueError("dependency entry is not an object")
+ components = (
+ dependency.get("dependency_type"),
+ dependency.get("dependency_id"),
+ dependency.get("source"),
+ dependency.get("target"),
+ dependency.get("label"),
+ )
+ return json.dumps(components, ensure_ascii=False, separators=(",", ":"))
+
+
+def _collect_changes(
+ before: Any,
+ after: Any,
+ *,
+ path: tuple[str, ...],
+ collector: _ChangeCollector,
+) -> None:
+ if before is _MISSING and after is _MISSING:
+ return
+ if isinstance(before, Mapping) and isinstance(after, Mapping):
+ # The walk shape never depends on ``include_values``: authorization decides what each
+ # change carries, not which changes exist, so ``max_changes`` means one thing and a
+ # caller allowed values never receives a less complete change set than a redacted one.
+ if _is_task_mapping_path(path):
+ _collect_task_changes(before, after, path=path, collector=collector)
+ return
+ if _is_task_group_mapping_path(path):
+ _collect_public_field_changes(
+ before,
+ after,
+ path=path,
+ public_fields=_DIFF_V1_PUBLIC_TASK_GROUP_FIELDS,
+ collector=collector,
+ )
+ return
+ if not _should_recurse_mapping(path):
+ if not _is_json_equal(before, after):
+ collector.add(path=path, operation="changed", before=before, after=after)
+ return
+ keys = sorted({str(key) for key in before} | {str(key) for key in after})
+ for key in keys:
+ before_value = before.get(key, _MISSING)
+ after_value = after.get(key, _MISSING)
+ _collect_changes(before_value, after_value, path=path + (key,), collector=collector)
+ if collector.is_truncated:
+ return
+ return
+
+ if isinstance(before, list) and isinstance(after, list):
+ if (
+ _is_task_group_child_path(path)
+ and len(before) == len(after) == 2
+ and before[0] == after[0] == "taskgroup"
+ and isinstance(before[1], Mapping)
+ and isinstance(after[1], Mapping)
+ ):
+ _collect_changes(before[1], after[1], path=path + ("1",), collector=collector)
+ return
+ if not _is_json_equal(before, after):
+ collector.add(path=path, operation="changed", before=before, after=after)
+ return
+
+ if before is _MISSING:
+ collector.add(path=path, operation="added", before=_MISSING, after=after)
+ elif after is _MISSING:
+ collector.add(path=path, operation="removed", before=before, after=_MISSING)
+ elif not _is_json_equal(before, after):
+ collector.add(path=path, operation="changed", before=before, after=after)
+
+
+def _collect_task_changes(
+ before: Mapping[str, Any],
+ after: Mapping[str, Any],
+ *,
+ path: tuple[str, ...],
+ collector: _ChangeCollector,
+) -> None:
+ if len(path) == 3:
+ _collect_changes(
+ before.get("__type", _MISSING),
+ after.get("__type", _MISSING),
+ path=path + ("__type",),
+ collector=collector,
+ )
+ if collector.is_truncated:
+ return
+ before, after = before["__var"], after["__var"]
+ public_fields = _DIFF_V1_PUBLIC_TASK_FIELDS - {"__type"}
+ else:
+ public_fields = _DIFF_V1_PUBLIC_PARTIAL_TASK_FIELDS
+ _collect_public_field_changes(before, after, path=path, public_fields=public_fields, collector=collector)
+
+
+def _collect_public_field_changes(
+ before: Mapping[str, Any],
+ after: Mapping[str, Any],
+ *,
+ path: tuple[str, ...],
+ public_fields: frozenset[str] | set[str],
+ collector: _ChangeCollector,
+) -> None:
+ """Walk allowlisted fields one by one and report everything else as a single change."""
+ keys = {str(key) for key in before} | {str(key) for key in after}
+ for key in sorted(keys & public_fields):
+ _collect_changes(
+ before.get(key, _MISSING),
+ after.get(key, _MISSING),
+ path=path + (key,),
+ collector=collector,
+ )
+ if collector.is_truncated:
+ return
+
+ before_custom_fields = {key: before[key] for key in before if key not in public_fields}
+ after_custom_fields = {key: after[key] for key in after if key not in public_fields}
+ if not _is_json_equal(before_custom_fields, after_custom_fields):
+ collector.add(
+ path=path + (_CUSTOM_TASK_FIELDS_PATH_COMPONENT,),
+ operation="changed",
+ before=before_custom_fields,
+ after=after_custom_fields,
+ )
+
+
+def _is_task_mapping_path(path: tuple[str, ...]) -> bool:
+ return path[:2] == ("dag", "tasks") and (
+ len(path) == 3 or (len(path) == 4 and path[3] == "partial_kwargs")
+ )
+
+
+def _is_task_group_mapping_path(path: tuple[str, ...]) -> bool:
+ index = _get_task_group_field_index(path)
+ return index is not None and len(path) == index
+
+
+def _should_recurse_mapping(path: tuple[str, ...]) -> bool:
+ if path in _RECURSIVE_MAPPING_PATHS:
+ return True
+ index = _get_task_group_field_index(path)
+ return index is not None and path[index:] == ("children",)
+
+
+def _get_task_group_field_index(path: tuple[str, ...]) -> int | None:
+ if path[:2] != ("dag", "task_group"):
+ return None
+ index = 2
+ while len(path) >= index + 3 and path[index] == "children" and path[index + 2] == "1":
+ index += 3
+ return index
+
+
+def _is_task_group_child_path(path: tuple[str, ...]) -> bool:
+ index = _get_task_group_field_index(path)
+ return index is not None and len(path) == index + 2 and path[index] == "children"
+
+
+def _get_public_path(path: tuple[str, ...]) -> tuple[str, ...]:
+ if path[:2] == ("dag", "task_group"):
+ public_path = list(path)
+ index = 2
+ while len(path) >= index + 2 and path[index] == "children":
+ public_path[index + 1] = "*"
+ if len(path) < index + 3 or path[index + 2] != "1":
+ return tuple(public_path)
+ index += 3
+ if len(path) > index and path[index] not in _DIFF_V1_PUBLIC_TASK_GROUP_FIELDS:
+ public_path[index] = "custom_fields"
+ return tuple(public_path)
+ if len(path) >= 3 and path[:2] in _KEYED_COLLECTION_PATHS:
+ path = (*path[:2], "*", *path[3:])
+ if len(path) >= 4 and path[:2] == ("dag", "tasks"):
+ field_index = 4 if path[3] == "partial_kwargs" and len(path) >= 5 else 3
+ public_fields = (
+ _DIFF_V1_PUBLIC_PARTIAL_TASK_FIELDS if field_index == 4 else _DIFF_V1_PUBLIC_TASK_FIELDS
+ )
+ if path[field_index] not in public_fields:
+ return (*path[:field_index], _CUSTOM_TASK_FIELDS_PATH_COMPONENT)
+ return path
+
+
+def _format_path(path: tuple[str, ...]) -> str:
+ return "/" + "/".join(component.replace("~", "~0").replace("/", "~1") for component in path)
+
+
+class _JsonEncodingError(TypeError):
+ """Distinguish unsupported JSON values from comparison implementation errors."""
+
+
+def _serialize_canonical_json(value: Any) -> str:
+ try:
+ return json.dumps(value, ensure_ascii=False, separators=(",", ":"), sort_keys=True)
+ except TypeError as error:
+ raise _JsonEncodingError from error
+
+
+def _is_json_equal(before: Any, after: Any) -> bool:
+ # Compare canonical JSON, not Python ==, so a JSON type change (1 vs 1.0, True vs 1,
+ # False vs 0) registers as a change instead of collapsing under Python equality.
+ return _serialize_canonical_json(before) == _serialize_canonical_json(after)
+
+
+def _get_digest(value: Any) -> str:
+ return f"sha256:{hashlib.sha256(_serialize_canonical_json(value).encode()).hexdigest()}"
Review Comment:
## CodeQL / Use of a broken or weak cryptographic hashing algorithm on sensitive data
[Sensitive data (password)](1) is used in a hashing algorithm (SHA256) that is insecure for password hashing, since it is not a computationally expensive hash function.
[Show more details](https://github.com/apache/airflow/security/code-scanning/655)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add serialized Dag version diff engine [airflow]
Posted by "kaxil (via GitHub)" <gi...@apache.org>.
kaxil commented on code in PR #69864:
URL: https://github.com/apache/airflow/pull/69864#discussion_r4008627236
##########
airflow-core/src/airflow/serialization/dag_version_diff.py:
##########
@@ -0,0 +1,898 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""Observed-state diffs for serialized Dag payloads."""
+
+from __future__ import annotations
+
+import copy
+import hashlib
+import json
+from collections.abc import Callable, Mapping
+from datetime import timedelta
+from enum import Enum
+from typing import Any, Literal
+
+import structlog
+
+from airflow.serialization.definitions.baseoperator import SerializedBaseOperator
+from airflow.serialization.definitions.mappedoperator import SerializedMappedOperator
+from airflow.serialization.serialized_objects import (
+ _DAG_CALLBACK_FIELDS,
+ _OPERATOR_TIMEDELTA_FIELDS,
+ DagSerialization,
+ OperatorSerialization,
+)
+
+log = structlog.get_logger(__name__)
+
+DIFF_SCHEMA_VERSION = 1
+DEFAULT_MAX_CHANGES = 500
+MAX_ALLOWED_CHANGES = 5000
+SUPPORTED_SERIALIZED_DAG_SCHEMA_VERSIONS = frozenset((1, 2, 3))
+
+_ORDER_INSENSITIVE_LIST_PATHS = {
+ ("dag", "tags"),
+ ("dag", "allowed_run_types"),
+}
+_KEYED_COLLECTION_PATHS = {
+ ("dag", "tasks"),
+ ("dag", "dag_dependencies"),
+ *_ORDER_INSENSITIVE_LIST_PATHS,
+}
+_CUSTOM_TASK_FIELDS_PATH_COMPONENT = "custom_fields"
+# This allowlist is part of diff schema v1. Serializer schema changes must not
+# silently change the paths visible to callers of the diff API.
+_DIFF_V1_PUBLIC_TASK_FIELDS = frozenset(
Review Comment:
This allowlist drops `python_callable_name`, which is the one live field the new `_upgrade_encoded_operator` split exists to preserve. `_preprocess_encoded_operator` strips exactly two fields, `python_callable_name` ("Only serves to detect function name changes") and `label` ("Shouldn't be set anymore"), and line 452 calls the new upgrade-only variant rather than the preprocess one so that signal survives into the payload. The allowlist then buries it in `custom_fields`. I checked in breeze: two versions of one Dag, same `task_id`, a `@task` function renamed from `extract_from_postgres` to `extract_from_snowflake`, `python_callable_name` the only key that differs. The diff reports one change, `/dag/tasks/*/custom_fields`, and never names the field, so a reviewer comparing two versions cannot see that a task's callable was swapped.
Five more emitted fields sit outside the allowlist the same way: `expand_input`, `op_kwargs_expand_input`, `resources`, `run_as_user`, `email` (`_operator_name` and `has_retry_policy` too). Changing `run_as_user` from `alice` to `root`, or `resources` from 1 cpu to 8, each reports a single `/dag/tasks/*/custom_fields`, both changed together are indistinguishable from either alone, and a mapped `.expand(bash_command=[...])` going from two commands to three reports only `custom_fields`, so what a mapped task expands over never shows up.
The parity test can't catch any of these, because `definitions.operator` is `additionalProperties: true`, so a field the serializer writes and the schema doesn't declare still passes set equality against `properties`. `_DIFF_V1_PUBLIC_TASK_GROUP_FIELDS` already carries `expand_input` for that reason, with a comment on its own parity test saying so, so this was closed on the group side and missed here. Pinning the task test to the key set of a real `to_dict()` payload would catch all of them; `get_serialized_fields()` alone misses `python_callable_name` and names `operator_extra_links` by its attribute rather than its serialized key `_operator_extra_links`.
##########
airflow-core/src/airflow/serialization/dag_version_diff.py:
##########
@@ -0,0 +1,898 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""Observed-state diffs for serialized Dag payloads."""
+
+from __future__ import annotations
+
+import copy
+import hashlib
+import json
+from collections.abc import Callable, Mapping
+from datetime import timedelta
+from enum import Enum
+from typing import Any, Literal
+
+import structlog
+
+from airflow.serialization.definitions.baseoperator import SerializedBaseOperator
+from airflow.serialization.definitions.mappedoperator import SerializedMappedOperator
+from airflow.serialization.serialized_objects import (
+ _DAG_CALLBACK_FIELDS,
+ _OPERATOR_TIMEDELTA_FIELDS,
+ DagSerialization,
+ OperatorSerialization,
+)
+
+log = structlog.get_logger(__name__)
+
+DIFF_SCHEMA_VERSION = 1
+DEFAULT_MAX_CHANGES = 500
+MAX_ALLOWED_CHANGES = 5000
+SUPPORTED_SERIALIZED_DAG_SCHEMA_VERSIONS = frozenset((1, 2, 3))
+
+_ORDER_INSENSITIVE_LIST_PATHS = {
+ ("dag", "tags"),
+ ("dag", "allowed_run_types"),
+}
+_KEYED_COLLECTION_PATHS = {
+ ("dag", "tasks"),
+ ("dag", "dag_dependencies"),
+ *_ORDER_INSENSITIVE_LIST_PATHS,
+}
+_CUSTOM_TASK_FIELDS_PATH_COMPONENT = "custom_fields"
+# This allowlist is part of diff schema v1. Serializer schema changes must not
+# silently change the paths visible to callers of the diff API.
+_DIFF_V1_PUBLIC_TASK_FIELDS = frozenset(
+ {
+ "__type",
+ "_disallow_kwargs_override",
+ "_expand_input_attr",
+ "_is_mapped",
+ "_is_sensor",
+ "_logger_name",
+ "_needs_expansion",
+ "_operator_extra_links",
+ "_task_display_name",
+ "_task_module",
+ "allow_nested_operators",
+ "depends_on_past",
+ "do_xcom_push",
+ "doc",
+ "doc_json",
+ "doc_md",
+ "doc_rst",
+ "doc_yaml",
+ "downstream_task_ids",
+ "email_on_failure",
+ "email_on_retry",
+ "end_date",
+ "execution_timeout",
+ "executor",
+ "executor_config",
+ "has_on_execute_callback",
+ "has_on_failure_callback",
+ "has_on_retry_callback",
+ "has_on_skipped_callback",
+ "has_on_success_callback",
+ "ignore_first_depends_on_past",
+ "inlets",
+ "is_setup",
+ "is_teardown",
+ "map_index_template",
+ "max_active_tis_per_dag",
+ "max_active_tis_per_dagrun",
+ "max_retry_delay",
+ "multiple_outputs",
+ "on_failure_fail_dagrun",
+ "outlets",
+ "owner",
+ "params",
+ "partial_kwargs",
+ "pool",
+ "pool_slots",
+ "priority_weight",
+ "queue",
+ "render_template_as_native_obj",
+ "retries",
+ "retry_delay",
+ "retry_exponential_backoff",
+ "start_date",
+ "start_from_trigger",
+ "start_trigger_args",
+ "task_id",
+ "task_type",
+ "template_ext",
+ "template_fields",
+ "template_fields_renderers",
+ "trigger_rule",
+ "ui_color",
+ "ui_fgcolor",
+ "wait_for_downstream",
+ "wait_for_past_depends_before_skipping",
+ "weight_rule",
+ }
+)
+_DIFF_V1_REDACTED_SCHEMA_TASK_FIELDS = frozenset({"_arg_bindings"})
+# _get_category classifies task fields with these; every name must be a public task field or the
+# entry is unreachable, since a non-public field is aggregated under custom_fields before lookup.
+_DIFF_V1_TASK_ASSET_FIELDS = frozenset({"inlets", "outlets"})
+_DIFF_V1_TASK_PARAM_FIELDS = frozenset({"params"})
+_DIFF_V1_TASK_DEPENDENCY_FIELDS = frozenset({"downstream_task_ids"})
+_DIFF_V1_TASK_METADATA_FIELDS = frozenset(
+ {
+ "doc",
+ "doc_json",
+ "doc_md",
+ "doc_rst",
+ "doc_yaml",
+ "owner",
+ "ui_color",
+ "ui_fgcolor",
+ "_task_display_name",
+ "task_display_name",
+ }
+)
+_DIFF_V1_PUBLIC_PARTIAL_TASK_FIELDS = _DIFF_V1_PUBLIC_TASK_FIELDS | {"task_display_name"}
+# Classify every Dag schema field explicitly so new fields require a policy decision.
+_DIFF_V1_DAG_FIELD_CATEGORIES = {
+ "_concurrency": "schedule",
+ "_processor_dags_folder": "provenance",
+ "access_control": "authorization",
+ "allowed_run_types": "schedule",
+ "bundle_name": "provenance",
+ "catchup": "schedule",
+ "dag_dependencies": "dependency",
+ "dag_display_name": "metadata",
+ "dag_id": "metadata",
+ "dagrun_timeout": "schedule",
+ "deadline": "deadline",
+ "default_args": "param",
+ "description": "metadata",
+ "disable_bundle_versioning": "task",
+ "doc_md": "metadata",
+ "edge_info": "metadata",
+ "end_date": "schedule",
+ "fail_fast": "schedule",
+ "fileloc": "provenance",
+ "has_on_failure_callback": "callback",
+ "has_on_success_callback": "callback",
+ "is_paused_upon_creation": "schedule",
+ "max_active_runs": "schedule",
+ "max_active_tasks": "schedule",
+ "max_consecutive_failed_dag_runs": "schedule",
+ "owner_links": "metadata",
+ "params": "param",
+ "relative_fileloc": "provenance",
+ "render_template_as_native_obj": "task",
+ "rerun_with_latest_version": "task",
+ "start_date": "schedule",
+ "tags": "metadata",
+ "task_group": "task",
+ "tasks": "task",
+ "timetable": "schedule",
+ "timezone": "schedule",
+}
+_DIFF_V1_LEGACY_DAG_FIELD_CATEGORIES = {
+ "fail_stop": "schedule",
+ "on_failure_callback": "callback",
+ "on_success_callback": "callback",
+ "schedule": "schedule",
+ "schedule_interval": "schedule",
+}
+_RECURSIVE_MAPPING_PATHS = {
+ (),
+ ("dag",),
+ ("provenance",),
+ *_KEYED_COLLECTION_PATHS,
+}
+_DIFF_V1_TASK_GROUP_METADATA_FIELDS = frozenset(
+ {"group_display_name", "tooltip", "doc_md", "ui_color", "ui_fgcolor"}
+)
+_DIFF_V1_PUBLIC_TASK_GROUP_FIELDS = _DIFF_V1_TASK_GROUP_METADATA_FIELDS | {
+ "_group_id",
+ "prefix_group_id",
+ "children",
+ "upstream_group_ids",
+ "downstream_group_ids",
+ "upstream_task_ids",
+ "downstream_task_ids",
+ "expand_input",
+ "is_mapped",
+}
+
+
+def build_unavailable_dag_diff(
+ *,
+ base_data: dict[str, Any] | None,
+ target_data: dict[str, Any] | None,
+ reason: str,
+) -> dict[str, Any]:
+ """Report a known unavailable reason without comparing the stored payloads."""
+ return _mark_unavailable(
+ _build_diff_result(_get_schema_version(base_data), _get_schema_version(target_data)), reason
+ )
+
+
+def build_serialized_dag_diff(
+ *,
+ base_data: dict[str, Any] | None,
+ target_data: dict[str, Any] | None,
+ base_provenance: Mapping[str, Any] | None = None,
+ target_provenance: Mapping[str, Any] | None = None,
+ include_values: bool = False,
+ max_changes: int = DEFAULT_MAX_CHANGES,
+) -> dict[str, Any]:
+ """
+ Build a bounded, deterministic diff from two stored serialized Dag payloads.
+
+ Raw values, digests, and value-derived path components are returned only when
+ ``include_values`` is true. Callers must authorize disclosure of the entire
+ serialized payload, including access-control role names and permission mappings,
+ before enabling it.
+ """
+ validate_max_changes(max_changes)
+
+ base_schema_version = _get_schema_version(base_data)
+ target_schema_version = _get_schema_version(target_data)
+ result = _build_diff_result(base_schema_version, target_schema_version)
+
+ if base_data is None or target_data is None:
+ return _mark_unavailable(result, "serialized_dag_missing")
+
+ if base_schema_version is None or target_schema_version is None:
+ return _mark_unavailable(result, "serialized_dag_schema_version_missing")
+
+ unsupported_versions = [
+ version
+ for version in (base_schema_version, target_schema_version)
+ if version not in SUPPORTED_SERIALIZED_DAG_SCHEMA_VERSIONS
+ ]
+ if unsupported_versions:
+ return _mark_unavailable(
+ result, f"unsupported_serialized_dag_schema_version:{unsupported_versions[0]}"
+ )
+
+ try:
+ base_document = _canonicalize_payload_v1(base_data)
+ target_document = _canonicalize_payload_v1(target_data)
+ base_document["provenance"] = _canonicalize_value(dict(base_provenance or {}), path=("provenance",))
+ target_document["provenance"] = _canonicalize_value(
+ dict(target_provenance or {}), path=("provenance",)
+ )
+ except (AttributeError, KeyError, OverflowError, TypeError, ValueError) as error:
+ log.warning(
+ "Serialized Dag diff canonicalization failed",
+ error_type=type(error).__name__,
+ base_schema_version=base_schema_version,
+ target_schema_version=target_schema_version,
+ )
+ return _mark_unavailable(result, "serialized_dag_canonicalization_failed")
+
+ collector = _ChangeCollector(max_changes=max_changes, include_values=include_values)
+ try:
+ _collect_changes(base_document, target_document, path=(), collector=collector)
+ except _JsonEncodingError:
+ log.warning(
+ "Serialized Dag diff JSON encoding failed",
+ base_schema_version=base_schema_version,
+ target_schema_version=target_schema_version,
+ )
+ return _mark_unavailable(result, "serialized_dag_json_encoding_failed")
+
+ result["changes"] = collector.changes
+ result["truncated"] = collector.is_truncated
+ if include_values:
+ result["values"] = {"status": "available"}
+ return result
+
+
+class _ChangeCollector:
+ def __init__(self, *, max_changes: int, include_values: bool) -> None:
+ self.changes: list[dict[str, Any]] = []
+ self.count = 0
+ self.max_changes = max_changes
+ self.include_values = include_values
+
+ @property
+ def is_truncated(self) -> bool:
+ return self.count > self.max_changes
+
+ def add(
+ self,
+ *,
+ path: tuple[str, ...],
+ operation: Literal["added", "removed", "changed"],
+ before: Any,
+ after: Any,
+ ) -> None:
+ self.count += 1
+ if len(self.changes) >= self.max_changes:
+ return
+
+ public_path = _get_public_path(path)
+ category = _get_category(public_path)
+ change: dict[str, Any] = {
+ "path": _format_path(path if self.include_values else public_path),
Review Comment:
Redacting in the path rather than in the walk lets the default response spend its whole budget on identical records. On a 600-task Dag where every task's `retries` and `start_date` changed, `max_changes=500` returns 500 change objects holding only 3 distinct values (250x `/dag/tasks/*/retries`, 249x `/dag/tasks/*/start_date`, 1x `/dag/start_date`) with `truncated: true`, while the same call with values authorized returns 500 distinct ones. Raising `max_changes` recovers nothing, because the duplicates scale with task count rather than with content.
This predates last round's walk-shape change (f755bc09 gave 2 distinct records out of 500), so it isn't a regression. But the endpoint in the next PR defaults to this mode, so is it worth collapsing equal public paths into one record carrying an occurrence count, so the default answer reads as 250 tasks changing `retries` instead of one object repeated 250 times?
##########
airflow-core/src/airflow/serialization/dag_version_diff.py:
##########
@@ -0,0 +1,898 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""Observed-state diffs for serialized Dag payloads."""
+
+from __future__ import annotations
+
+import copy
+import hashlib
+import json
+from collections.abc import Callable, Mapping
+from datetime import timedelta
+from enum import Enum
+from typing import Any, Literal
+
+import structlog
+
+from airflow.serialization.definitions.baseoperator import SerializedBaseOperator
+from airflow.serialization.definitions.mappedoperator import SerializedMappedOperator
+from airflow.serialization.serialized_objects import (
+ _DAG_CALLBACK_FIELDS,
+ _OPERATOR_TIMEDELTA_FIELDS,
+ DagSerialization,
+ OperatorSerialization,
+)
+
+log = structlog.get_logger(__name__)
+
+DIFF_SCHEMA_VERSION = 1
+DEFAULT_MAX_CHANGES = 500
+MAX_ALLOWED_CHANGES = 5000
+SUPPORTED_SERIALIZED_DAG_SCHEMA_VERSIONS = frozenset((1, 2, 3))
+
+_ORDER_INSENSITIVE_LIST_PATHS = {
+ ("dag", "tags"),
+ ("dag", "allowed_run_types"),
+}
+_KEYED_COLLECTION_PATHS = {
+ ("dag", "tasks"),
+ ("dag", "dag_dependencies"),
+ *_ORDER_INSENSITIVE_LIST_PATHS,
+}
+_CUSTOM_TASK_FIELDS_PATH_COMPONENT = "custom_fields"
+# This allowlist is part of diff schema v1. Serializer schema changes must not
+# silently change the paths visible to callers of the diff API.
+_DIFF_V1_PUBLIC_TASK_FIELDS = frozenset(
+ {
+ "__type",
+ "_disallow_kwargs_override",
+ "_expand_input_attr",
+ "_is_mapped",
+ "_is_sensor",
+ "_logger_name",
+ "_needs_expansion",
+ "_operator_extra_links",
+ "_task_display_name",
+ "_task_module",
+ "allow_nested_operators",
+ "depends_on_past",
+ "do_xcom_push",
+ "doc",
+ "doc_json",
+ "doc_md",
+ "doc_rst",
+ "doc_yaml",
+ "downstream_task_ids",
+ "email_on_failure",
+ "email_on_retry",
+ "end_date",
+ "execution_timeout",
+ "executor",
+ "executor_config",
+ "has_on_execute_callback",
+ "has_on_failure_callback",
+ "has_on_retry_callback",
+ "has_on_skipped_callback",
+ "has_on_success_callback",
+ "ignore_first_depends_on_past",
+ "inlets",
+ "is_setup",
+ "is_teardown",
+ "map_index_template",
+ "max_active_tis_per_dag",
+ "max_active_tis_per_dagrun",
+ "max_retry_delay",
+ "multiple_outputs",
+ "on_failure_fail_dagrun",
+ "outlets",
+ "owner",
+ "params",
+ "partial_kwargs",
+ "pool",
+ "pool_slots",
+ "priority_weight",
+ "queue",
+ "render_template_as_native_obj",
+ "retries",
+ "retry_delay",
+ "retry_exponential_backoff",
+ "start_date",
+ "start_from_trigger",
+ "start_trigger_args",
+ "task_id",
+ "task_type",
+ "template_ext",
+ "template_fields",
+ "template_fields_renderers",
+ "trigger_rule",
+ "ui_color",
+ "ui_fgcolor",
+ "wait_for_downstream",
+ "wait_for_past_depends_before_skipping",
+ "weight_rule",
+ }
+)
+_DIFF_V1_REDACTED_SCHEMA_TASK_FIELDS = frozenset({"_arg_bindings"})
+# _get_category classifies task fields with these; every name must be a public task field or the
+# entry is unreachable, since a non-public field is aggregated under custom_fields before lookup.
+_DIFF_V1_TASK_ASSET_FIELDS = frozenset({"inlets", "outlets"})
+_DIFF_V1_TASK_PARAM_FIELDS = frozenset({"params"})
+_DIFF_V1_TASK_DEPENDENCY_FIELDS = frozenset({"downstream_task_ids"})
+_DIFF_V1_TASK_METADATA_FIELDS = frozenset(
+ {
+ "doc",
+ "doc_json",
+ "doc_md",
+ "doc_rst",
+ "doc_yaml",
+ "owner",
+ "ui_color",
+ "ui_fgcolor",
+ "_task_display_name",
+ "task_display_name",
+ }
+)
+_DIFF_V1_PUBLIC_PARTIAL_TASK_FIELDS = _DIFF_V1_PUBLIC_TASK_FIELDS | {"task_display_name"}
+# Classify every Dag schema field explicitly so new fields require a policy decision.
+_DIFF_V1_DAG_FIELD_CATEGORIES = {
+ "_concurrency": "schedule",
+ "_processor_dags_folder": "provenance",
+ "access_control": "authorization",
+ "allowed_run_types": "schedule",
+ "bundle_name": "provenance",
+ "catchup": "schedule",
+ "dag_dependencies": "dependency",
+ "dag_display_name": "metadata",
+ "dag_id": "metadata",
+ "dagrun_timeout": "schedule",
+ "deadline": "deadline",
+ "default_args": "param",
+ "description": "metadata",
+ "disable_bundle_versioning": "task",
+ "doc_md": "metadata",
+ "edge_info": "metadata",
+ "end_date": "schedule",
+ "fail_fast": "schedule",
+ "fileloc": "provenance",
+ "has_on_failure_callback": "callback",
+ "has_on_success_callback": "callback",
+ "is_paused_upon_creation": "schedule",
+ "max_active_runs": "schedule",
+ "max_active_tasks": "schedule",
+ "max_consecutive_failed_dag_runs": "schedule",
+ "owner_links": "metadata",
+ "params": "param",
+ "relative_fileloc": "provenance",
+ "render_template_as_native_obj": "task",
+ "rerun_with_latest_version": "task",
+ "start_date": "schedule",
+ "tags": "metadata",
+ "task_group": "task",
+ "tasks": "task",
+ "timetable": "schedule",
+ "timezone": "schedule",
+}
+_DIFF_V1_LEGACY_DAG_FIELD_CATEGORIES = {
+ "fail_stop": "schedule",
+ "on_failure_callback": "callback",
+ "on_success_callback": "callback",
+ "schedule": "schedule",
+ "schedule_interval": "schedule",
+}
+_RECURSIVE_MAPPING_PATHS = {
+ (),
+ ("dag",),
+ ("provenance",),
+ *_KEYED_COLLECTION_PATHS,
+}
+_DIFF_V1_TASK_GROUP_METADATA_FIELDS = frozenset(
+ {"group_display_name", "tooltip", "doc_md", "ui_color", "ui_fgcolor"}
+)
+_DIFF_V1_PUBLIC_TASK_GROUP_FIELDS = _DIFF_V1_TASK_GROUP_METADATA_FIELDS | {
+ "_group_id",
+ "prefix_group_id",
+ "children",
+ "upstream_group_ids",
+ "downstream_group_ids",
+ "upstream_task_ids",
+ "downstream_task_ids",
+ "expand_input",
+ "is_mapped",
+}
+
+
+def build_unavailable_dag_diff(
+ *,
+ base_data: dict[str, Any] | None,
+ target_data: dict[str, Any] | None,
+ reason: str,
+) -> dict[str, Any]:
+ """Report a known unavailable reason without comparing the stored payloads."""
+ return _mark_unavailable(
+ _build_diff_result(_get_schema_version(base_data), _get_schema_version(target_data)), reason
+ )
+
+
+def build_serialized_dag_diff(
+ *,
+ base_data: dict[str, Any] | None,
+ target_data: dict[str, Any] | None,
+ base_provenance: Mapping[str, Any] | None = None,
+ target_provenance: Mapping[str, Any] | None = None,
+ include_values: bool = False,
+ max_changes: int = DEFAULT_MAX_CHANGES,
+) -> dict[str, Any]:
+ """
+ Build a bounded, deterministic diff from two stored serialized Dag payloads.
+
+ Raw values, digests, and value-derived path components are returned only when
+ ``include_values`` is true. Callers must authorize disclosure of the entire
+ serialized payload, including access-control role names and permission mappings,
+ before enabling it.
+ """
+ validate_max_changes(max_changes)
+
+ base_schema_version = _get_schema_version(base_data)
+ target_schema_version = _get_schema_version(target_data)
+ result = _build_diff_result(base_schema_version, target_schema_version)
+
+ if base_data is None or target_data is None:
+ return _mark_unavailable(result, "serialized_dag_missing")
+
+ if base_schema_version is None or target_schema_version is None:
+ return _mark_unavailable(result, "serialized_dag_schema_version_missing")
+
+ unsupported_versions = [
+ version
+ for version in (base_schema_version, target_schema_version)
+ if version not in SUPPORTED_SERIALIZED_DAG_SCHEMA_VERSIONS
+ ]
+ if unsupported_versions:
+ return _mark_unavailable(
+ result, f"unsupported_serialized_dag_schema_version:{unsupported_versions[0]}"
+ )
+
+ try:
+ base_document = _canonicalize_payload_v1(base_data)
+ target_document = _canonicalize_payload_v1(target_data)
+ base_document["provenance"] = _canonicalize_value(dict(base_provenance or {}), path=("provenance",))
+ target_document["provenance"] = _canonicalize_value(
+ dict(target_provenance or {}), path=("provenance",)
+ )
+ except (AttributeError, KeyError, OverflowError, TypeError, ValueError) as error:
Review Comment:
`RecursionError` subclasses `RuntimeError`, so it falls outside this tuple and isn't `_JsonEncodingError` either, which makes it the one canonicalization failure that escapes instead of becoming `mode: "unavailable"`.
What makes a payload writable but not readable here is structural rather than a curiosity about pathological configs. `BaseOperator.__deepcopy__` in the Task SDK opens with `sys.setrecursionlimit(5000)`, a process-global raise that fires in the dag-processor as soon as any operator is deep-copied during parsing, while `SerializedBaseOperator` has no such hook, so the API server that will serve this diff stays at the default 1000. Measured in breeze: the parser stores an `executor_config` nested 2491 deep once that limit is raised, and 491 deep without it, while `build_serialized_dag_diff` starts raising at 244, dropping to 219 and 194 with 100 and 200 extra frames already on the stack, which a real request has. So rows the parser wrote happily come back as a 500 on the next PR's endpoint rather than the unavailable result every other failure mode here produces. Worth covering the `_collect_changes` call below the same way, since `json.dumps` recurses in C and `_serialize_canonical
_json` only converts `TypeError`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add serialized Dag version diff engine [airflow]
Posted by "ephraimbuddy (via GitHub)" <gi...@apache.org>.
ephraimbuddy commented on PR #69864:
URL: https://github.com/apache/airflow/pull/69864#issuecomment-4971387362
Extracted from https://github.com/apache/airflow/pull/69818 and reworked
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add serialized Dag version diff engine [airflow]
Posted by "kaxil (via GitHub)" <gi...@apache.org>.
kaxil commented on code in PR #69864:
URL: https://github.com/apache/airflow/pull/69864#discussion_r3989433174
##########
airflow-core/src/airflow/serialization/dag_version_diff.py:
##########
@@ -0,0 +1,752 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""Observed-state diffs for serialized Dag payloads."""
+
+from __future__ import annotations
+
+import copy
+import hashlib
+import json
+from collections.abc import Callable, Mapping
+from datetime import timedelta
+from enum import Enum
+from typing import Any, Literal
+
+import structlog
+
+from airflow.serialization.definitions.baseoperator import SerializedBaseOperator
+from airflow.serialization.definitions.mappedoperator import SerializedMappedOperator
+from airflow.serialization.serialized_objects import (
+ _DAG_CALLBACK_FIELDS,
+ _OPERATOR_TIMEDELTA_FIELDS,
+ DagSerialization,
+ OperatorSerialization,
+)
+
+log = structlog.get_logger(__name__)
+
+DIFF_SCHEMA_VERSION = 1
+DEFAULT_MAX_CHANGES = 500
+MAX_ALLOWED_CHANGES = 5000
+SUPPORTED_SERIALIZED_DAG_SCHEMA_VERSIONS = frozenset((1, 2, 3))
Review Comment:
This is the only schema-derived constant in the module without a parity test, while `_DIFF_V1_PUBLIC_TASK_FIELDS`, `_DIFF_V1_DAG_FIELD_CATEGORIES`, `_DAG_CALLBACK_FIELDS` and `_OPERATOR_TIMEDELTA_FIELDS` each have one. The day `SERIALIZER_VERSION` becomes 4, every stored row comes back `unsupported_serialized_dag_schema_version:4` for every Dag in the deployment and the suite stays green. `frozenset(range(1, DagSerialization.SERIALIZER_VERSION + 1))`, or a sixth parity test alongside the others, would close it.
##########
airflow-core/src/airflow/serialization/dag_version_diff.py:
##########
@@ -0,0 +1,752 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""Observed-state diffs for serialized Dag payloads."""
+
+from __future__ import annotations
+
+import copy
+import hashlib
+import json
+from collections.abc import Callable, Mapping
+from datetime import timedelta
+from enum import Enum
+from typing import Any, Literal
+
+import structlog
+
+from airflow.serialization.definitions.baseoperator import SerializedBaseOperator
+from airflow.serialization.definitions.mappedoperator import SerializedMappedOperator
+from airflow.serialization.serialized_objects import (
+ _DAG_CALLBACK_FIELDS,
+ _OPERATOR_TIMEDELTA_FIELDS,
+ DagSerialization,
+ OperatorSerialization,
+)
+
+log = structlog.get_logger(__name__)
+
+DIFF_SCHEMA_VERSION = 1
+DEFAULT_MAX_CHANGES = 500
+MAX_ALLOWED_CHANGES = 5000
+SUPPORTED_SERIALIZED_DAG_SCHEMA_VERSIONS = frozenset((1, 2, 3))
+
+_ORDER_INSENSITIVE_LIST_PATHS = {
+ ("dag", "tags"),
+ ("dag", "allowed_run_types"),
+}
+_KEYED_COLLECTION_PATHS = {
+ ("dag", "tasks"),
+ ("dag", "dag_dependencies"),
+ *_ORDER_INSENSITIVE_LIST_PATHS,
+}
+_CUSTOM_TASK_FIELDS_PATH_COMPONENT = "custom_fields"
+# This allowlist is part of diff schema v1. Serializer schema changes must not
+# silently change the paths visible to callers of the diff API.
+_DIFF_V1_PUBLIC_TASK_FIELDS = frozenset(
+ {
+ "__type",
+ "_disallow_kwargs_override",
+ "_expand_input_attr",
+ "_is_mapped",
+ "_is_sensor",
+ "_logger_name",
+ "_needs_expansion",
+ "_operator_extra_links",
+ "_task_display_name",
+ "_task_module",
+ "allow_nested_operators",
+ "depends_on_past",
+ "do_xcom_push",
+ "doc",
+ "doc_json",
+ "doc_md",
+ "doc_rst",
+ "doc_yaml",
+ "downstream_task_ids",
+ "email_on_failure",
+ "email_on_retry",
+ "end_date",
+ "execution_timeout",
+ "executor",
+ "executor_config",
+ "has_on_execute_callback",
+ "has_on_failure_callback",
+ "has_on_retry_callback",
+ "has_on_skipped_callback",
+ "has_on_success_callback",
+ "ignore_first_depends_on_past",
+ "inlets",
+ "is_setup",
+ "is_teardown",
+ "map_index_template",
+ "max_active_tis_per_dag",
+ "max_active_tis_per_dagrun",
+ "max_retry_delay",
+ "multiple_outputs",
+ "on_failure_fail_dagrun",
+ "outlets",
+ "owner",
+ "params",
+ "partial_kwargs",
+ "pool",
+ "pool_slots",
+ "priority_weight",
+ "queue",
+ "render_template_as_native_obj",
+ "retries",
+ "retry_delay",
+ "retry_exponential_backoff",
+ "start_date",
+ "start_from_trigger",
+ "start_trigger_args",
+ "task_id",
+ "task_type",
+ "template_ext",
+ "template_fields",
+ "template_fields_renderers",
+ "trigger_rule",
+ "ui_color",
+ "ui_fgcolor",
+ "wait_for_downstream",
+ "wait_for_past_depends_before_skipping",
+ "weight_rule",
+ }
+)
+_DIFF_V1_REDACTED_SCHEMA_TASK_FIELDS = frozenset({"_arg_bindings"})
+_DIFF_V1_PUBLIC_PARTIAL_TASK_FIELDS = _DIFF_V1_PUBLIC_TASK_FIELDS | {"task_display_name"}
+# Classify every Dag schema field explicitly so new fields require a policy decision.
+_DIFF_V1_DAG_FIELD_CATEGORIES = {
+ "_concurrency": "schedule",
+ "_processor_dags_folder": "provenance",
+ "access_control": "authorization",
+ "allowed_run_types": "schedule",
+ "bundle_name": "provenance",
+ "catchup": "schedule",
+ "dag_dependencies": "dependency",
+ "dag_display_name": "metadata",
+ "dag_id": "metadata",
+ "dagrun_timeout": "schedule",
+ "deadline": "deadline",
+ "default_args": "param",
+ "description": "metadata",
+ "disable_bundle_versioning": "task",
+ "doc_md": "metadata",
+ "edge_info": "metadata",
+ "end_date": "schedule",
+ "fail_fast": "schedule",
+ "fileloc": "provenance",
+ "has_on_failure_callback": "callback",
+ "has_on_success_callback": "callback",
+ "is_paused_upon_creation": "schedule",
+ "max_active_runs": "schedule",
+ "max_active_tasks": "schedule",
+ "max_consecutive_failed_dag_runs": "schedule",
+ "owner_links": "metadata",
+ "params": "param",
+ "relative_fileloc": "provenance",
+ "render_template_as_native_obj": "task",
+ "rerun_with_latest_version": "task",
+ "start_date": "schedule",
+ "tags": "metadata",
+ "task_group": "task",
+ "tasks": "task",
+ "timetable": "schedule",
+ "timezone": "schedule",
+}
+_DIFF_V1_LEGACY_DAG_FIELD_CATEGORIES = {
+ "fail_stop": "schedule",
+ "on_failure_callback": "callback",
+ "on_success_callback": "callback",
+ "schedule": "schedule",
+ "schedule_interval": "schedule",
+}
+_REDACTED_RECURSIVE_MAPPING_PATHS = {
+ (),
+ ("dag",),
+ ("dag", "task_group"),
+ ("provenance",),
+ *_KEYED_COLLECTION_PATHS,
+}
+
+
+def build_unavailable_dag_diff(
+ *,
+ base_data: dict[str, Any] | None,
+ target_data: dict[str, Any] | None,
+ reason: str,
+) -> dict[str, Any]:
+ """Report a known unavailable reason without comparing the stored payloads."""
+ return _mark_unavailable(
+ _build_diff_result(_get_schema_version(base_data), _get_schema_version(target_data)), reason
+ )
+
+
+def build_serialized_dag_diff(
+ *,
+ base_data: dict[str, Any] | None,
+ target_data: dict[str, Any] | None,
+ base_provenance: Mapping[str, Any] | None = None,
+ target_provenance: Mapping[str, Any] | None = None,
+ include_values: bool = False,
+ max_changes: int = DEFAULT_MAX_CHANGES,
+) -> dict[str, Any]:
+ """
+ Build a bounded, deterministic diff from two stored serialized Dag payloads.
+
+ Raw values, digests, and value-derived path components are returned only when
+ ``include_values`` is true. Callers must authorize disclosure of the entire
+ serialized payload, including access-control role names and permission mappings,
+ before enabling it.
+ """
+ validate_max_changes(max_changes)
+
+ base_schema_version = _get_schema_version(base_data)
+ target_schema_version = _get_schema_version(target_data)
+ result = _build_diff_result(base_schema_version, target_schema_version)
+
+ if base_data is None or target_data is None:
+ return _mark_unavailable(result, "serialized_dag_missing")
+
+ if base_schema_version is None or target_schema_version is None:
+ return _mark_unavailable(result, "serialized_dag_schema_version_missing")
+
+ unsupported_versions = [
+ version
+ for version in (base_schema_version, target_schema_version)
+ if version not in SUPPORTED_SERIALIZED_DAG_SCHEMA_VERSIONS
+ ]
+ if unsupported_versions:
+ return _mark_unavailable(
+ result, f"unsupported_serialized_dag_schema_version:{unsupported_versions[0]}"
+ )
+
+ try:
+ base_document = _canonicalize_payload_v1(base_data)
+ target_document = _canonicalize_payload_v1(target_data)
+ base_document["provenance"] = _canonicalize_value(dict(base_provenance or {}), path=("provenance",))
+ target_document["provenance"] = _canonicalize_value(
+ dict(target_provenance or {}), path=("provenance",)
+ )
+ except (AttributeError, KeyError, OverflowError, TypeError, ValueError) as error:
+ log.warning(
+ "Serialized Dag diff canonicalization failed",
+ error_type=type(error).__name__,
+ base_schema_version=base_schema_version,
+ target_schema_version=target_schema_version,
+ )
+ return _mark_unavailable(result, "serialized_dag_canonicalization_failed")
+
+ collector = _ChangeCollector(max_changes=max_changes, include_values=include_values)
+ try:
+ _collect_changes(base_document, target_document, path=(), collector=collector)
+ except _JsonEncodingError:
+ log.warning(
+ "Serialized Dag diff JSON encoding failed",
+ base_schema_version=base_schema_version,
+ target_schema_version=target_schema_version,
+ )
+ return _mark_unavailable(result, "serialized_dag_json_encoding_failed")
+
+ result["changes"] = collector.changes
+ result["truncated"] = collector.is_truncated
+ return result
+
+
+class _ChangeCollector:
+ def __init__(self, *, max_changes: int, include_values: bool) -> None:
+ self.changes: list[dict[str, Any]] = []
+ self.count = 0
+ self.max_changes = max_changes
+ self.include_values = include_values
+
+ @property
+ def is_truncated(self) -> bool:
+ return self.count > self.max_changes
+
+ def add(
+ self,
+ *,
+ path: tuple[str, ...],
+ operation: Literal["added", "removed", "changed"],
+ before: Any,
+ after: Any,
+ ) -> None:
+ self.count += 1
+ if len(self.changes) >= self.max_changes:
+ return
+
+ public_path = _get_public_path(path)
+ category = _get_category(public_path)
+ change: dict[str, Any] = {
+ "path": _format_path(path if self.include_values else public_path),
+ "operation": operation,
+ "category": category,
+ "impact": _get_impact(category),
+ }
+ if self.include_values:
+ change["before_digest"] = None if before is _MISSING else _get_digest(before)
+ change["after_digest"] = None if after is _MISSING else _get_digest(after)
+ if before is not _MISSING:
+ change["before_value"] = before
+ if after is not _MISSING:
+ change["after_value"] = after
+ self.changes.append(change)
+
+
+_MISSING = object()
+
+
+def validate_max_changes(max_changes: int) -> None:
+ if max_changes < 1:
+ raise ValueError("max_changes must be a positive integer")
+ if max_changes > MAX_ALLOWED_CHANGES:
+ raise ValueError(f"max_changes must not exceed {MAX_ALLOWED_CHANGES}")
+
+
+def _get_schema_version(data: Mapping[str, Any] | None) -> int | None:
+ if not isinstance(data, Mapping):
+ return None
+ version = data.get("__version")
+ return version if isinstance(version, int) and not isinstance(version, bool) else None
+
+
+def _mark_unavailable(result: dict[str, Any], reason: str) -> dict[str, Any]:
+ result["mode"] = "unavailable"
+ result["unavailable_reason"] = reason
+ return result
+
+
+def _build_diff_result(base_schema_version: int | None, target_schema_version: int | None) -> dict[str, Any]:
+ return {
+ "diff_schema_version": DIFF_SCHEMA_VERSION,
+ "serialized_dag_schema_versions": {
+ "base": base_schema_version,
+ "target": target_schema_version,
+ },
+ "mode": "observed_state",
+ "changes": [],
+ "truncated": False,
+ }
+
+
+def _canonicalize_payload_v1(data: dict[str, Any]) -> dict[str, Any]:
+ payload = copy.deepcopy(data)
+ version = _get_schema_version(payload)
+ if version is None:
+ raise ValueError("missing or invalid __version")
+ if version == 1:
+ DagSerialization.conversion_v1_to_v2(payload)
+ DagSerialization.conversion_v2_to_v3(payload)
+ elif version == 2:
+ DagSerialization.conversion_v2_to_v3(payload)
+ if not isinstance(payload.get("dag"), Mapping):
+ raise ValueError("missing dag object")
+ dag_defaults = {
+ field: value
+ for field, value in DagSerialization.get_schema_defaults("dag").items()
+ # Dag callback flags are enabled by their presence, even when their value is false.
+ if field not in _DAG_CALLBACK_FIELDS
+ }
+ payload["dag"] = {**dag_defaults, **payload["dag"]}
+ for field in _DAG_CALLBACK_FIELDS & payload["dag"].keys():
+ payload["dag"][field] = True
+ _apply_task_defaults(payload)
+ payload.pop("__version", None)
+ return _canonicalize_value(payload, path=())
+
+
+def _apply_task_defaults(payload: dict[str, Any]) -> None:
+ client_defaults = payload.pop("client_defaults", None)
+ if client_defaults is None:
+ client_defaults = {}
+ if not isinstance(client_defaults, Mapping):
+ raise ValueError("client_defaults is not an object")
+
+ task_defaults = client_defaults.get("tasks", {})
+ if not isinstance(task_defaults, Mapping):
+ raise ValueError("client_defaults.tasks is not an object")
+
+ schema_defaults = DagSerialization.get_schema_defaults("operator")
+ partial_fields = (
+ SerializedBaseOperator.get_serialized_fields() - SerializedMappedOperator.get_serialized_fields()
+ )
+ # A mapped task resolves these through partial_kwargs, so their defaults have to be applied
+ # there and encoded like any other partial value rather than left at the outer level.
+ partial_schema_defaults = {
+ field: value for field, value in schema_defaults.items() if field in partial_fields
+ }
+ outer_schema_defaults = {
+ field: value for field, value in schema_defaults.items() if field not in partial_fields
+ }
+ tasks = payload["dag"].get("tasks", [])
+ if not isinstance(tasks, list):
+ raise ValueError("dag.tasks is not a list")
+ for task in tasks:
+ if not isinstance(task, dict) or not isinstance(task.get("__var"), Mapping):
+ raise ValueError("task entry is not an object")
+ encoded_task = OperatorSerialization._apply_defaults_to_encoded_op(
+ dict(task["__var"]), dict(client_defaults)
+ )
+ upgraded_task = OperatorSerialization._upgrade_encoded_operator(encoded_task)
+ if upgraded_task.get("_is_mapped"):
+ task_data = {**outer_schema_defaults, **upgraded_task}
+ partial_kwargs = task_data.get("partial_kwargs", {})
+ if not isinstance(partial_kwargs, Mapping):
+ raise ValueError("partial_kwargs is not an object")
+ # populate_operator only folds client defaults into partial_kwargs when the payload
+ # carries the key, so an absent one leaves the top-level value as the effective value.
+ effective_partial_kwargs = (
+ {field: _encode_partial_field_value(field, value) for field, value in task_defaults.items()}
+ if "partial_kwargs" in task_data
+ else {}
+ )
+ for field, value in partial_kwargs.items():
+ if isinstance(value, Mapping) and "__type" in value and "__var" in value:
+ effective_partial_kwargs[field] = value
+ else:
+ effective_partial_kwargs[field] = _encode_partial_field_value(field, value)
+ # Match populate_operator: partial values take precedence over outer task defaults.
+ template_fields = task_data.get("template_fields", [])
+ for field in partial_fields & task_data.keys():
+ value = task_data.pop(field)
+ # Outer fields are already encoded unless template handling bypasses deserialization.
+ if field in template_fields:
+ value = _encode_json_value(value)
+ effective_partial_kwargs.setdefault(field, value)
+ for field, value in partial_schema_defaults.items():
+ effective_partial_kwargs.setdefault(field, _encode_partial_field_value(field, value))
+ task_data["partial_kwargs"] = effective_partial_kwargs
+ _normalize_retry_backoff(effective_partial_kwargs)
+ else:
+ task_data = {**schema_defaults, **upgraded_task}
+ _normalize_retry_backoff(task_data)
+ task["__var"] = task_data
+
+
+def _encode_partial_field_value(field: str, value: Any) -> Any:
+ # Only partial kwargs and client defaults use field-specific timedelta deserialization.
+ if field in _OPERATOR_TIMEDELTA_FIELDS and value is not None:
+ return {"__type": "timedelta", "__var": value}
+ return _encode_json_value(value)
+
+
+def _encode_json_value(value: Any) -> Any:
+ """Encode plain JSON without invoking object serializers."""
+ if isinstance(value, Mapping):
+ return {"__type": "dict", "__var": {key: _encode_json_value(item) for key, item in value.items()}}
+ if isinstance(value, list):
+ return [_encode_json_value(item) for item in value]
+ return value
+
+
+def _normalize_retry_backoff(task_fields: dict[str, Any]) -> None:
+ if "retry_exponential_backoff" in task_fields:
+ value = task_fields["retry_exponential_backoff"]
+ task_fields["retry_exponential_backoff"] = 2.0 if value is True else float(value)
+
+
+def _canonicalize_value(value: Any, *, path: tuple[str, ...]) -> Any:
+ if isinstance(value, Mapping):
+ if path == ("dag", "deadline"):
+ if value.get("__type") == "deadline_alert":
+ value = value["__var"]
+ value = {"name": None, **value}
+ interval = value.get("interval")
+ if isinstance(interval, (int, float)) and not isinstance(interval, bool):
+ # Diff schema v1 uses the SDK's version-2 timedelta encoding for legacy seconds.
+ value["interval"] = {
+ "__classname__": "datetime.timedelta",
+ "__version__": 2,
+ "__data__": timedelta(seconds=interval).total_seconds(),
+ }
+ return {
+ canonical_key: _canonicalize_value(item, path=path + (canonical_key,))
+ for canonical_key, item in (
+ (_canonicalize_mapping_key(key), item)
+ for key, item in sorted(value.items(), key=lambda item: _canonicalize_mapping_key(item[0]))
+ )
+ }
+ if isinstance(value, list):
+ canonical_values = [_canonicalize_value(item, path=path) for item in value]
+ if path == ("dag", "tasks"):
+ return _canonicalize_keyed_list(canonical_values, _get_task_id, path)
+ if path == ("dag", "dag_dependencies"):
+ return _canonicalize_keyed_list(canonical_values, _get_dependency_key, path)
+ if path in _ORDER_INSENSITIVE_LIST_PATHS:
+ return _canonicalize_keyed_list(canonical_values, _get_string_key, path)
+ return canonical_values
+ return value
+
+
+def _canonicalize_mapping_key(key: Any) -> str:
+ if isinstance(key, str) and isinstance(key, Enum):
+ return key.value
+ return str(key)
+
+
+def _canonicalize_keyed_list(
+ values: list[Any], key_getter: Callable[[Any], str], path: tuple[str, ...]
+) -> dict[str, Any]:
+ keyed_values: dict[str, Any] = {}
+ for value in values:
+ key = key_getter(value)
+ if key in keyed_values:
+ if path in _ORDER_INSENSITIVE_LIST_PATHS or (
+ path == ("dag", "dag_dependencies") and keyed_values[key] == value
+ ):
+ continue
+ raise ValueError(f"duplicate key {key!r} in /{'/'.join(path)}")
+ canonical_value = value
+ if path == ("dag", "tasks") and isinstance(value, Mapping) and "__var" in value:
+ task_value = dict(value["__var"])
+ if "__type" in value:
+ task_value["__type"] = value["__type"]
+ canonical_value = task_value
+ keyed_values[key] = canonical_value
+ return {key: keyed_values[key] for key in sorted(keyed_values)}
+
+
+def _get_task_id(task: Any) -> str:
+ if not isinstance(task, Mapping):
+ raise ValueError("task entry is not an object")
+ task_data = task.get("__var", task)
+ task_id = task_data.get("task_id") if isinstance(task_data, Mapping) else None
+ if not isinstance(task_id, str):
+ raise ValueError("task entry has no task_id")
+ return task_id
+
+
+def _get_string_key(value: Any) -> str:
+ if not isinstance(value, str):
+ raise ValueError("collection entry is not a string")
+ return value
+
+
+def _get_dependency_key(dependency: Any) -> str:
+ if not isinstance(dependency, Mapping):
+ raise ValueError("dependency entry is not an object")
+ components = (
+ dependency.get("dependency_type"),
+ dependency.get("dependency_id"),
+ dependency.get("source"),
+ dependency.get("target"),
+ dependency.get("label"),
+ )
+ return json.dumps(components, ensure_ascii=False, separators=(",", ":"))
+
+
+def _collect_changes(
+ before: Any,
+ after: Any,
+ *,
+ path: tuple[str, ...],
+ collector: _ChangeCollector,
+) -> None:
+ if isinstance(before, Mapping) and isinstance(after, Mapping):
+ if not collector.include_values and _is_task_mapping_path(path):
+ _collect_redacted_task_changes(before, after, path=path, collector=collector)
+ return
+ if not collector.include_values and not _should_recurse_redacted_mapping(path):
Review Comment:
These two guards gate how deep the walk goes, not just whether values get attached, so `include_values` changes the change set itself rather than only redacting it. On two versions of a 100-task Dag where every task's 20-key `executor_config` changed, `max_changes=500` gives 100 changes and `truncated=False` when redacted, but 500 changes, `truncated=True` and only 25 of the 100 tasks represented once values are authorized; the other 75 never appear at any path. So `max_changes` means two different things depending on a flag the client also sets, and the more-authorized caller gets the less complete answer, which contradicts `get_diff`'s docstring on line 271 ("leaving the structural diff available in redacted form"). Cheaper to settle here than after the endpoint in the next PR fixes it into a public API: compute the change set once with the redacted recursion shape, then attach values and digests to those same records when authorized.
##########
airflow-core/src/airflow/serialization/dag_version_diff.py:
##########
@@ -0,0 +1,752 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""Observed-state diffs for serialized Dag payloads."""
+
+from __future__ import annotations
+
+import copy
+import hashlib
+import json
+from collections.abc import Callable, Mapping
+from datetime import timedelta
+from enum import Enum
+from typing import Any, Literal
+
+import structlog
+
+from airflow.serialization.definitions.baseoperator import SerializedBaseOperator
+from airflow.serialization.definitions.mappedoperator import SerializedMappedOperator
+from airflow.serialization.serialized_objects import (
+ _DAG_CALLBACK_FIELDS,
+ _OPERATOR_TIMEDELTA_FIELDS,
+ DagSerialization,
+ OperatorSerialization,
+)
+
+log = structlog.get_logger(__name__)
+
+DIFF_SCHEMA_VERSION = 1
+DEFAULT_MAX_CHANGES = 500
+MAX_ALLOWED_CHANGES = 5000
+SUPPORTED_SERIALIZED_DAG_SCHEMA_VERSIONS = frozenset((1, 2, 3))
+
+_ORDER_INSENSITIVE_LIST_PATHS = {
+ ("dag", "tags"),
+ ("dag", "allowed_run_types"),
+}
+_KEYED_COLLECTION_PATHS = {
+ ("dag", "tasks"),
+ ("dag", "dag_dependencies"),
+ *_ORDER_INSENSITIVE_LIST_PATHS,
+}
+_CUSTOM_TASK_FIELDS_PATH_COMPONENT = "custom_fields"
+# This allowlist is part of diff schema v1. Serializer schema changes must not
+# silently change the paths visible to callers of the diff API.
+_DIFF_V1_PUBLIC_TASK_FIELDS = frozenset(
+ {
+ "__type",
+ "_disallow_kwargs_override",
+ "_expand_input_attr",
+ "_is_mapped",
+ "_is_sensor",
+ "_logger_name",
+ "_needs_expansion",
+ "_operator_extra_links",
+ "_task_display_name",
+ "_task_module",
+ "allow_nested_operators",
+ "depends_on_past",
+ "do_xcom_push",
+ "doc",
+ "doc_json",
+ "doc_md",
+ "doc_rst",
+ "doc_yaml",
+ "downstream_task_ids",
+ "email_on_failure",
+ "email_on_retry",
+ "end_date",
+ "execution_timeout",
+ "executor",
+ "executor_config",
+ "has_on_execute_callback",
+ "has_on_failure_callback",
+ "has_on_retry_callback",
+ "has_on_skipped_callback",
+ "has_on_success_callback",
+ "ignore_first_depends_on_past",
+ "inlets",
+ "is_setup",
+ "is_teardown",
+ "map_index_template",
+ "max_active_tis_per_dag",
+ "max_active_tis_per_dagrun",
+ "max_retry_delay",
+ "multiple_outputs",
+ "on_failure_fail_dagrun",
+ "outlets",
+ "owner",
+ "params",
+ "partial_kwargs",
+ "pool",
+ "pool_slots",
+ "priority_weight",
+ "queue",
+ "render_template_as_native_obj",
+ "retries",
+ "retry_delay",
+ "retry_exponential_backoff",
+ "start_date",
+ "start_from_trigger",
+ "start_trigger_args",
+ "task_id",
+ "task_type",
+ "template_ext",
+ "template_fields",
+ "template_fields_renderers",
+ "trigger_rule",
+ "ui_color",
+ "ui_fgcolor",
+ "wait_for_downstream",
+ "wait_for_past_depends_before_skipping",
+ "weight_rule",
+ }
+)
+_DIFF_V1_REDACTED_SCHEMA_TASK_FIELDS = frozenset({"_arg_bindings"})
+_DIFF_V1_PUBLIC_PARTIAL_TASK_FIELDS = _DIFF_V1_PUBLIC_TASK_FIELDS | {"task_display_name"}
+# Classify every Dag schema field explicitly so new fields require a policy decision.
+_DIFF_V1_DAG_FIELD_CATEGORIES = {
+ "_concurrency": "schedule",
+ "_processor_dags_folder": "provenance",
+ "access_control": "authorization",
+ "allowed_run_types": "schedule",
+ "bundle_name": "provenance",
+ "catchup": "schedule",
+ "dag_dependencies": "dependency",
+ "dag_display_name": "metadata",
+ "dag_id": "metadata",
+ "dagrun_timeout": "schedule",
+ "deadline": "deadline",
+ "default_args": "param",
+ "description": "metadata",
+ "disable_bundle_versioning": "task",
+ "doc_md": "metadata",
+ "edge_info": "metadata",
+ "end_date": "schedule",
+ "fail_fast": "schedule",
+ "fileloc": "provenance",
+ "has_on_failure_callback": "callback",
+ "has_on_success_callback": "callback",
+ "is_paused_upon_creation": "schedule",
+ "max_active_runs": "schedule",
+ "max_active_tasks": "schedule",
+ "max_consecutive_failed_dag_runs": "schedule",
+ "owner_links": "metadata",
+ "params": "param",
+ "relative_fileloc": "provenance",
+ "render_template_as_native_obj": "task",
+ "rerun_with_latest_version": "task",
+ "start_date": "schedule",
+ "tags": "metadata",
+ "task_group": "task",
+ "tasks": "task",
+ "timetable": "schedule",
+ "timezone": "schedule",
+}
+_DIFF_V1_LEGACY_DAG_FIELD_CATEGORIES = {
+ "fail_stop": "schedule",
+ "on_failure_callback": "callback",
+ "on_success_callback": "callback",
+ "schedule": "schedule",
+ "schedule_interval": "schedule",
+}
+_REDACTED_RECURSIVE_MAPPING_PATHS = {
+ (),
+ ("dag",),
+ ("dag", "task_group"),
+ ("provenance",),
+ *_KEYED_COLLECTION_PATHS,
+}
+
+
+def build_unavailable_dag_diff(
+ *,
+ base_data: dict[str, Any] | None,
+ target_data: dict[str, Any] | None,
+ reason: str,
+) -> dict[str, Any]:
+ """Report a known unavailable reason without comparing the stored payloads."""
+ return _mark_unavailable(
+ _build_diff_result(_get_schema_version(base_data), _get_schema_version(target_data)), reason
+ )
+
+
+def build_serialized_dag_diff(
+ *,
+ base_data: dict[str, Any] | None,
+ target_data: dict[str, Any] | None,
+ base_provenance: Mapping[str, Any] | None = None,
+ target_provenance: Mapping[str, Any] | None = None,
+ include_values: bool = False,
+ max_changes: int = DEFAULT_MAX_CHANGES,
+) -> dict[str, Any]:
+ """
+ Build a bounded, deterministic diff from two stored serialized Dag payloads.
+
+ Raw values, digests, and value-derived path components are returned only when
+ ``include_values`` is true. Callers must authorize disclosure of the entire
+ serialized payload, including access-control role names and permission mappings,
+ before enabling it.
+ """
+ validate_max_changes(max_changes)
+
+ base_schema_version = _get_schema_version(base_data)
+ target_schema_version = _get_schema_version(target_data)
+ result = _build_diff_result(base_schema_version, target_schema_version)
+
+ if base_data is None or target_data is None:
+ return _mark_unavailable(result, "serialized_dag_missing")
+
+ if base_schema_version is None or target_schema_version is None:
+ return _mark_unavailable(result, "serialized_dag_schema_version_missing")
+
+ unsupported_versions = [
+ version
+ for version in (base_schema_version, target_schema_version)
+ if version not in SUPPORTED_SERIALIZED_DAG_SCHEMA_VERSIONS
+ ]
+ if unsupported_versions:
+ return _mark_unavailable(
+ result, f"unsupported_serialized_dag_schema_version:{unsupported_versions[0]}"
+ )
+
+ try:
+ base_document = _canonicalize_payload_v1(base_data)
+ target_document = _canonicalize_payload_v1(target_data)
+ base_document["provenance"] = _canonicalize_value(dict(base_provenance or {}), path=("provenance",))
+ target_document["provenance"] = _canonicalize_value(
+ dict(target_provenance or {}), path=("provenance",)
+ )
+ except (AttributeError, KeyError, OverflowError, TypeError, ValueError) as error:
+ log.warning(
+ "Serialized Dag diff canonicalization failed",
+ error_type=type(error).__name__,
+ base_schema_version=base_schema_version,
+ target_schema_version=target_schema_version,
+ )
+ return _mark_unavailable(result, "serialized_dag_canonicalization_failed")
+
+ collector = _ChangeCollector(max_changes=max_changes, include_values=include_values)
+ try:
+ _collect_changes(base_document, target_document, path=(), collector=collector)
+ except _JsonEncodingError:
+ log.warning(
+ "Serialized Dag diff JSON encoding failed",
+ base_schema_version=base_schema_version,
+ target_schema_version=target_schema_version,
+ )
+ return _mark_unavailable(result, "serialized_dag_json_encoding_failed")
+
+ result["changes"] = collector.changes
+ result["truncated"] = collector.is_truncated
+ return result
+
+
+class _ChangeCollector:
+ def __init__(self, *, max_changes: int, include_values: bool) -> None:
+ self.changes: list[dict[str, Any]] = []
+ self.count = 0
+ self.max_changes = max_changes
+ self.include_values = include_values
+
+ @property
+ def is_truncated(self) -> bool:
+ return self.count > self.max_changes
+
+ def add(
+ self,
+ *,
+ path: tuple[str, ...],
+ operation: Literal["added", "removed", "changed"],
+ before: Any,
+ after: Any,
+ ) -> None:
+ self.count += 1
+ if len(self.changes) >= self.max_changes:
+ return
+
+ public_path = _get_public_path(path)
+ category = _get_category(public_path)
+ change: dict[str, Any] = {
+ "path": _format_path(path if self.include_values else public_path),
+ "operation": operation,
+ "category": category,
+ "impact": _get_impact(category),
+ }
+ if self.include_values:
+ change["before_digest"] = None if before is _MISSING else _get_digest(before)
+ change["after_digest"] = None if after is _MISSING else _get_digest(after)
+ if before is not _MISSING:
+ change["before_value"] = before
+ if after is not _MISSING:
+ change["after_value"] = after
+ self.changes.append(change)
+
+
+_MISSING = object()
+
+
+def validate_max_changes(max_changes: int) -> None:
+ if max_changes < 1:
+ raise ValueError("max_changes must be a positive integer")
+ if max_changes > MAX_ALLOWED_CHANGES:
+ raise ValueError(f"max_changes must not exceed {MAX_ALLOWED_CHANGES}")
+
+
+def _get_schema_version(data: Mapping[str, Any] | None) -> int | None:
+ if not isinstance(data, Mapping):
+ return None
+ version = data.get("__version")
+ return version if isinstance(version, int) and not isinstance(version, bool) else None
+
+
+def _mark_unavailable(result: dict[str, Any], reason: str) -> dict[str, Any]:
+ result["mode"] = "unavailable"
+ result["unavailable_reason"] = reason
+ return result
+
+
+def _build_diff_result(base_schema_version: int | None, target_schema_version: int | None) -> dict[str, Any]:
+ return {
+ "diff_schema_version": DIFF_SCHEMA_VERSION,
+ "serialized_dag_schema_versions": {
+ "base": base_schema_version,
+ "target": target_schema_version,
+ },
+ "mode": "observed_state",
+ "changes": [],
+ "truncated": False,
Review Comment:
`values` is attached by `DagVersion.get_diff`, not here, so both exported engine functions return a result without that key. The description says every returned result includes `values.status` and the comment in `get_diff` calls that read always safe, which holds for the model helper but not for anything calling `build_serialized_dag_diff` or `build_unavailable_dag_diff` directly, which is the natural route for the endpoint in the next PR. Emitting it from `_build_diff_result` would keep `diff_schema_version: 1` one shape across all three entry points.
##########
airflow-core/src/airflow/serialization/dag_version_diff.py:
##########
@@ -0,0 +1,752 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""Observed-state diffs for serialized Dag payloads."""
+
+from __future__ import annotations
+
+import copy
+import hashlib
+import json
+from collections.abc import Callable, Mapping
+from datetime import timedelta
+from enum import Enum
+from typing import Any, Literal
+
+import structlog
+
+from airflow.serialization.definitions.baseoperator import SerializedBaseOperator
+from airflow.serialization.definitions.mappedoperator import SerializedMappedOperator
+from airflow.serialization.serialized_objects import (
+ _DAG_CALLBACK_FIELDS,
+ _OPERATOR_TIMEDELTA_FIELDS,
+ DagSerialization,
+ OperatorSerialization,
+)
+
+log = structlog.get_logger(__name__)
+
+DIFF_SCHEMA_VERSION = 1
+DEFAULT_MAX_CHANGES = 500
+MAX_ALLOWED_CHANGES = 5000
+SUPPORTED_SERIALIZED_DAG_SCHEMA_VERSIONS = frozenset((1, 2, 3))
+
+_ORDER_INSENSITIVE_LIST_PATHS = {
+ ("dag", "tags"),
+ ("dag", "allowed_run_types"),
+}
+_KEYED_COLLECTION_PATHS = {
+ ("dag", "tasks"),
+ ("dag", "dag_dependencies"),
+ *_ORDER_INSENSITIVE_LIST_PATHS,
+}
+_CUSTOM_TASK_FIELDS_PATH_COMPONENT = "custom_fields"
+# This allowlist is part of diff schema v1. Serializer schema changes must not
+# silently change the paths visible to callers of the diff API.
+_DIFF_V1_PUBLIC_TASK_FIELDS = frozenset(
+ {
+ "__type",
+ "_disallow_kwargs_override",
+ "_expand_input_attr",
+ "_is_mapped",
+ "_is_sensor",
+ "_logger_name",
+ "_needs_expansion",
+ "_operator_extra_links",
+ "_task_display_name",
+ "_task_module",
+ "allow_nested_operators",
+ "depends_on_past",
+ "do_xcom_push",
+ "doc",
+ "doc_json",
+ "doc_md",
+ "doc_rst",
+ "doc_yaml",
+ "downstream_task_ids",
+ "email_on_failure",
+ "email_on_retry",
+ "end_date",
+ "execution_timeout",
+ "executor",
+ "executor_config",
+ "has_on_execute_callback",
+ "has_on_failure_callback",
+ "has_on_retry_callback",
+ "has_on_skipped_callback",
+ "has_on_success_callback",
+ "ignore_first_depends_on_past",
+ "inlets",
+ "is_setup",
+ "is_teardown",
+ "map_index_template",
+ "max_active_tis_per_dag",
+ "max_active_tis_per_dagrun",
+ "max_retry_delay",
+ "multiple_outputs",
+ "on_failure_fail_dagrun",
+ "outlets",
+ "owner",
+ "params",
+ "partial_kwargs",
+ "pool",
+ "pool_slots",
+ "priority_weight",
+ "queue",
+ "render_template_as_native_obj",
+ "retries",
+ "retry_delay",
+ "retry_exponential_backoff",
+ "start_date",
+ "start_from_trigger",
+ "start_trigger_args",
+ "task_id",
+ "task_type",
+ "template_ext",
+ "template_fields",
+ "template_fields_renderers",
+ "trigger_rule",
+ "ui_color",
+ "ui_fgcolor",
+ "wait_for_downstream",
+ "wait_for_past_depends_before_skipping",
+ "weight_rule",
+ }
+)
+_DIFF_V1_REDACTED_SCHEMA_TASK_FIELDS = frozenset({"_arg_bindings"})
+_DIFF_V1_PUBLIC_PARTIAL_TASK_FIELDS = _DIFF_V1_PUBLIC_TASK_FIELDS | {"task_display_name"}
+# Classify every Dag schema field explicitly so new fields require a policy decision.
+_DIFF_V1_DAG_FIELD_CATEGORIES = {
+ "_concurrency": "schedule",
+ "_processor_dags_folder": "provenance",
+ "access_control": "authorization",
+ "allowed_run_types": "schedule",
+ "bundle_name": "provenance",
+ "catchup": "schedule",
+ "dag_dependencies": "dependency",
+ "dag_display_name": "metadata",
+ "dag_id": "metadata",
+ "dagrun_timeout": "schedule",
+ "deadline": "deadline",
+ "default_args": "param",
+ "description": "metadata",
+ "disable_bundle_versioning": "task",
+ "doc_md": "metadata",
+ "edge_info": "metadata",
+ "end_date": "schedule",
+ "fail_fast": "schedule",
+ "fileloc": "provenance",
+ "has_on_failure_callback": "callback",
+ "has_on_success_callback": "callback",
+ "is_paused_upon_creation": "schedule",
+ "max_active_runs": "schedule",
+ "max_active_tasks": "schedule",
+ "max_consecutive_failed_dag_runs": "schedule",
+ "owner_links": "metadata",
+ "params": "param",
+ "relative_fileloc": "provenance",
+ "render_template_as_native_obj": "task",
+ "rerun_with_latest_version": "task",
+ "start_date": "schedule",
+ "tags": "metadata",
+ "task_group": "task",
+ "tasks": "task",
+ "timetable": "schedule",
+ "timezone": "schedule",
+}
+_DIFF_V1_LEGACY_DAG_FIELD_CATEGORIES = {
+ "fail_stop": "schedule",
+ "on_failure_callback": "callback",
+ "on_success_callback": "callback",
+ "schedule": "schedule",
+ "schedule_interval": "schedule",
+}
+_REDACTED_RECURSIVE_MAPPING_PATHS = {
+ (),
+ ("dag",),
+ ("dag", "task_group"),
+ ("provenance",),
+ *_KEYED_COLLECTION_PATHS,
+}
+
+
+def build_unavailable_dag_diff(
+ *,
+ base_data: dict[str, Any] | None,
+ target_data: dict[str, Any] | None,
+ reason: str,
+) -> dict[str, Any]:
+ """Report a known unavailable reason without comparing the stored payloads."""
+ return _mark_unavailable(
+ _build_diff_result(_get_schema_version(base_data), _get_schema_version(target_data)), reason
+ )
+
+
+def build_serialized_dag_diff(
+ *,
+ base_data: dict[str, Any] | None,
+ target_data: dict[str, Any] | None,
+ base_provenance: Mapping[str, Any] | None = None,
+ target_provenance: Mapping[str, Any] | None = None,
+ include_values: bool = False,
+ max_changes: int = DEFAULT_MAX_CHANGES,
+) -> dict[str, Any]:
+ """
+ Build a bounded, deterministic diff from two stored serialized Dag payloads.
+
+ Raw values, digests, and value-derived path components are returned only when
+ ``include_values`` is true. Callers must authorize disclosure of the entire
+ serialized payload, including access-control role names and permission mappings,
+ before enabling it.
+ """
+ validate_max_changes(max_changes)
+
+ base_schema_version = _get_schema_version(base_data)
+ target_schema_version = _get_schema_version(target_data)
+ result = _build_diff_result(base_schema_version, target_schema_version)
+
+ if base_data is None or target_data is None:
+ return _mark_unavailable(result, "serialized_dag_missing")
+
+ if base_schema_version is None or target_schema_version is None:
+ return _mark_unavailable(result, "serialized_dag_schema_version_missing")
+
+ unsupported_versions = [
+ version
+ for version in (base_schema_version, target_schema_version)
+ if version not in SUPPORTED_SERIALIZED_DAG_SCHEMA_VERSIONS
+ ]
+ if unsupported_versions:
+ return _mark_unavailable(
+ result, f"unsupported_serialized_dag_schema_version:{unsupported_versions[0]}"
+ )
+
+ try:
+ base_document = _canonicalize_payload_v1(base_data)
+ target_document = _canonicalize_payload_v1(target_data)
+ base_document["provenance"] = _canonicalize_value(dict(base_provenance or {}), path=("provenance",))
+ target_document["provenance"] = _canonicalize_value(
+ dict(target_provenance or {}), path=("provenance",)
+ )
+ except (AttributeError, KeyError, OverflowError, TypeError, ValueError) as error:
+ log.warning(
+ "Serialized Dag diff canonicalization failed",
+ error_type=type(error).__name__,
+ base_schema_version=base_schema_version,
+ target_schema_version=target_schema_version,
+ )
+ return _mark_unavailable(result, "serialized_dag_canonicalization_failed")
+
+ collector = _ChangeCollector(max_changes=max_changes, include_values=include_values)
+ try:
+ _collect_changes(base_document, target_document, path=(), collector=collector)
+ except _JsonEncodingError:
+ log.warning(
+ "Serialized Dag diff JSON encoding failed",
+ base_schema_version=base_schema_version,
+ target_schema_version=target_schema_version,
+ )
+ return _mark_unavailable(result, "serialized_dag_json_encoding_failed")
+
+ result["changes"] = collector.changes
+ result["truncated"] = collector.is_truncated
+ return result
+
+
+class _ChangeCollector:
+ def __init__(self, *, max_changes: int, include_values: bool) -> None:
+ self.changes: list[dict[str, Any]] = []
+ self.count = 0
+ self.max_changes = max_changes
+ self.include_values = include_values
+
+ @property
+ def is_truncated(self) -> bool:
+ return self.count > self.max_changes
+
+ def add(
+ self,
+ *,
+ path: tuple[str, ...],
+ operation: Literal["added", "removed", "changed"],
+ before: Any,
+ after: Any,
+ ) -> None:
+ self.count += 1
+ if len(self.changes) >= self.max_changes:
+ return
+
+ public_path = _get_public_path(path)
+ category = _get_category(public_path)
+ change: dict[str, Any] = {
+ "path": _format_path(path if self.include_values else public_path),
+ "operation": operation,
+ "category": category,
+ "impact": _get_impact(category),
+ }
+ if self.include_values:
+ change["before_digest"] = None if before is _MISSING else _get_digest(before)
+ change["after_digest"] = None if after is _MISSING else _get_digest(after)
+ if before is not _MISSING:
+ change["before_value"] = before
+ if after is not _MISSING:
+ change["after_value"] = after
+ self.changes.append(change)
+
+
+_MISSING = object()
+
+
+def validate_max_changes(max_changes: int) -> None:
+ if max_changes < 1:
+ raise ValueError("max_changes must be a positive integer")
+ if max_changes > MAX_ALLOWED_CHANGES:
+ raise ValueError(f"max_changes must not exceed {MAX_ALLOWED_CHANGES}")
+
+
+def _get_schema_version(data: Mapping[str, Any] | None) -> int | None:
+ if not isinstance(data, Mapping):
+ return None
+ version = data.get("__version")
+ return version if isinstance(version, int) and not isinstance(version, bool) else None
+
+
+def _mark_unavailable(result: dict[str, Any], reason: str) -> dict[str, Any]:
+ result["mode"] = "unavailable"
+ result["unavailable_reason"] = reason
+ return result
+
+
+def _build_diff_result(base_schema_version: int | None, target_schema_version: int | None) -> dict[str, Any]:
+ return {
+ "diff_schema_version": DIFF_SCHEMA_VERSION,
+ "serialized_dag_schema_versions": {
+ "base": base_schema_version,
+ "target": target_schema_version,
+ },
+ "mode": "observed_state",
+ "changes": [],
+ "truncated": False,
+ }
+
+
+def _canonicalize_payload_v1(data: dict[str, Any]) -> dict[str, Any]:
+ payload = copy.deepcopy(data)
+ version = _get_schema_version(payload)
+ if version is None:
+ raise ValueError("missing or invalid __version")
+ if version == 1:
+ DagSerialization.conversion_v1_to_v2(payload)
+ DagSerialization.conversion_v2_to_v3(payload)
+ elif version == 2:
+ DagSerialization.conversion_v2_to_v3(payload)
+ if not isinstance(payload.get("dag"), Mapping):
+ raise ValueError("missing dag object")
+ dag_defaults = {
+ field: value
+ for field, value in DagSerialization.get_schema_defaults("dag").items()
+ # Dag callback flags are enabled by their presence, even when their value is false.
+ if field not in _DAG_CALLBACK_FIELDS
+ }
+ payload["dag"] = {**dag_defaults, **payload["dag"]}
+ for field in _DAG_CALLBACK_FIELDS & payload["dag"].keys():
+ payload["dag"][field] = True
+ _apply_task_defaults(payload)
+ payload.pop("__version", None)
+ return _canonicalize_value(payload, path=())
+
+
+def _apply_task_defaults(payload: dict[str, Any]) -> None:
+ client_defaults = payload.pop("client_defaults", None)
+ if client_defaults is None:
+ client_defaults = {}
+ if not isinstance(client_defaults, Mapping):
+ raise ValueError("client_defaults is not an object")
+
+ task_defaults = client_defaults.get("tasks", {})
Review Comment:
The whole `client_defaults` section gets popped but only `tasks` is folded back in, so anything else in there is dropped before the comparison rather than rejected. Two payloads differing only in `client_defaults["dags"]["catchup"]` come back `observed_state` with `changes: []`. Given `client_defaults` is the growth mechanism v3 introduced, raising on unrecognised keys the way the two checks just above already raise on bad types would stop the diff going quiet the first time a new section lands.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add serialized Dag version diff engine [airflow]
Posted by "ephraimbuddy (via GitHub)" <gi...@apache.org>.
ephraimbuddy commented on code in PR #69864:
URL: https://github.com/apache/airflow/pull/69864#discussion_r4005652376
##########
airflow-core/src/airflow/serialization/dag_version_diff.py:
##########
@@ -0,0 +1,896 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""Observed-state diffs for serialized Dag payloads."""
+
+from __future__ import annotations
+
+import copy
+import hashlib
+import json
+from collections.abc import Callable, Mapping
+from datetime import timedelta
+from enum import Enum
+from typing import Any, Literal
+
+import structlog
+
+from airflow.serialization.definitions.baseoperator import SerializedBaseOperator
+from airflow.serialization.definitions.mappedoperator import SerializedMappedOperator
+from airflow.serialization.serialized_objects import (
+ _DAG_CALLBACK_FIELDS,
+ _OPERATOR_TIMEDELTA_FIELDS,
+ DagSerialization,
+ OperatorSerialization,
+)
+
+log = structlog.get_logger(__name__)
+
+DIFF_SCHEMA_VERSION = 1
+DEFAULT_MAX_CHANGES = 500
+MAX_ALLOWED_CHANGES = 5000
+SUPPORTED_SERIALIZED_DAG_SCHEMA_VERSIONS = frozenset((1, 2, 3))
+
+_ORDER_INSENSITIVE_LIST_PATHS = {
+ ("dag", "tags"),
+ ("dag", "allowed_run_types"),
+}
+_KEYED_COLLECTION_PATHS = {
+ ("dag", "tasks"),
+ ("dag", "dag_dependencies"),
+ *_ORDER_INSENSITIVE_LIST_PATHS,
+}
+_CUSTOM_TASK_FIELDS_PATH_COMPONENT = "custom_fields"
+# This allowlist is part of diff schema v1. Serializer schema changes must not
+# silently change the paths visible to callers of the diff API.
+_DIFF_V1_PUBLIC_TASK_FIELDS = frozenset(
+ {
+ "__type",
+ "_disallow_kwargs_override",
+ "_expand_input_attr",
+ "_is_mapped",
+ "_is_sensor",
+ "_logger_name",
+ "_needs_expansion",
+ "_operator_extra_links",
+ "_task_display_name",
+ "_task_module",
+ "allow_nested_operators",
+ "depends_on_past",
+ "do_xcom_push",
+ "doc",
+ "doc_json",
+ "doc_md",
+ "doc_rst",
+ "doc_yaml",
+ "downstream_task_ids",
+ "email_on_failure",
+ "email_on_retry",
+ "end_date",
+ "execution_timeout",
+ "executor",
+ "executor_config",
+ "has_on_execute_callback",
+ "has_on_failure_callback",
+ "has_on_retry_callback",
+ "has_on_skipped_callback",
+ "has_on_success_callback",
+ "ignore_first_depends_on_past",
+ "inlets",
+ "is_setup",
+ "is_teardown",
+ "map_index_template",
+ "max_active_tis_per_dag",
+ "max_active_tis_per_dagrun",
+ "max_retry_delay",
+ "multiple_outputs",
+ "on_failure_fail_dagrun",
+ "outlets",
+ "owner",
+ "params",
+ "partial_kwargs",
+ "pool",
+ "pool_slots",
+ "priority_weight",
+ "queue",
+ "render_template_as_native_obj",
+ "retries",
+ "retry_delay",
+ "retry_exponential_backoff",
+ "start_date",
+ "start_from_trigger",
+ "start_trigger_args",
+ "task_id",
+ "task_type",
+ "template_ext",
+ "template_fields",
+ "template_fields_renderers",
+ "trigger_rule",
+ "ui_color",
+ "ui_fgcolor",
+ "wait_for_downstream",
+ "wait_for_past_depends_before_skipping",
+ "weight_rule",
+ }
+)
+_DIFF_V1_REDACTED_SCHEMA_TASK_FIELDS = frozenset({"_arg_bindings"})
+# _get_category classifies task fields with these; every name must be a public task field or the
+# entry is unreachable, since a non-public field is aggregated under custom_fields before lookup.
+_DIFF_V1_TASK_ASSET_FIELDS = frozenset({"inlets", "outlets"})
+_DIFF_V1_TASK_PARAM_FIELDS = frozenset({"params"})
+_DIFF_V1_TASK_DEPENDENCY_FIELDS = frozenset({"downstream_task_ids"})
+_DIFF_V1_TASK_METADATA_FIELDS = frozenset(
+ {
+ "doc",
+ "doc_json",
+ "doc_md",
+ "doc_rst",
+ "doc_yaml",
+ "owner",
+ "ui_color",
+ "ui_fgcolor",
+ "_task_display_name",
+ "task_display_name",
+ }
+)
+_DIFF_V1_PUBLIC_PARTIAL_TASK_FIELDS = _DIFF_V1_PUBLIC_TASK_FIELDS | {"task_display_name"}
+# Classify every Dag schema field explicitly so new fields require a policy decision.
+_DIFF_V1_DAG_FIELD_CATEGORIES = {
+ "_concurrency": "schedule",
+ "_processor_dags_folder": "provenance",
+ "access_control": "authorization",
+ "allowed_run_types": "schedule",
+ "bundle_name": "provenance",
+ "catchup": "schedule",
+ "dag_dependencies": "dependency",
+ "dag_display_name": "metadata",
+ "dag_id": "metadata",
+ "dagrun_timeout": "schedule",
+ "deadline": "deadline",
+ "default_args": "param",
+ "description": "metadata",
+ "disable_bundle_versioning": "task",
+ "doc_md": "metadata",
+ "edge_info": "metadata",
+ "end_date": "schedule",
+ "fail_fast": "schedule",
+ "fileloc": "provenance",
+ "has_on_failure_callback": "callback",
+ "has_on_success_callback": "callback",
+ "is_paused_upon_creation": "schedule",
+ "max_active_runs": "schedule",
+ "max_active_tasks": "schedule",
+ "max_consecutive_failed_dag_runs": "schedule",
+ "owner_links": "metadata",
+ "params": "param",
+ "relative_fileloc": "provenance",
+ "render_template_as_native_obj": "task",
+ "rerun_with_latest_version": "task",
+ "start_date": "schedule",
+ "tags": "metadata",
+ "task_group": "task",
+ "tasks": "task",
+ "timetable": "schedule",
+ "timezone": "schedule",
+}
+_DIFF_V1_LEGACY_DAG_FIELD_CATEGORIES = {
+ "fail_stop": "schedule",
+ "on_failure_callback": "callback",
+ "on_success_callback": "callback",
+ "schedule": "schedule",
+ "schedule_interval": "schedule",
+}
+_RECURSIVE_MAPPING_PATHS = {
+ (),
+ ("dag",),
+ ("provenance",),
+ *_KEYED_COLLECTION_PATHS,
+}
+_DIFF_V1_TASK_GROUP_METADATA_FIELDS = frozenset(
+ {"group_display_name", "tooltip", "doc_md", "ui_color", "ui_fgcolor"}
+)
+_DIFF_V1_PUBLIC_TASK_GROUP_FIELDS = _DIFF_V1_TASK_GROUP_METADATA_FIELDS | {
+ "_group_id",
+ "prefix_group_id",
+ "children",
+ "upstream_group_ids",
+ "downstream_group_ids",
+ "upstream_task_ids",
+ "downstream_task_ids",
+ "expand_input",
+ "is_mapped",
+}
+
+
+def build_unavailable_dag_diff(
+ *,
+ base_data: dict[str, Any] | None,
+ target_data: dict[str, Any] | None,
+ reason: str,
+) -> dict[str, Any]:
+ """Report a known unavailable reason without comparing the stored payloads."""
+ return _mark_unavailable(
+ _build_diff_result(_get_schema_version(base_data), _get_schema_version(target_data)), reason
+ )
+
+
+def build_serialized_dag_diff(
+ *,
+ base_data: dict[str, Any] | None,
+ target_data: dict[str, Any] | None,
+ base_provenance: Mapping[str, Any] | None = None,
+ target_provenance: Mapping[str, Any] | None = None,
+ include_values: bool = False,
+ max_changes: int = DEFAULT_MAX_CHANGES,
+) -> dict[str, Any]:
+ """
+ Build a bounded, deterministic diff from two stored serialized Dag payloads.
+
+ Raw values, digests, and value-derived path components are returned only when
+ ``include_values`` is true. Callers must authorize disclosure of the entire
+ serialized payload, including access-control role names and permission mappings,
+ before enabling it.
+ """
+ validate_max_changes(max_changes)
+
+ base_schema_version = _get_schema_version(base_data)
+ target_schema_version = _get_schema_version(target_data)
+ result = _build_diff_result(base_schema_version, target_schema_version)
+
+ if base_data is None or target_data is None:
+ return _mark_unavailable(result, "serialized_dag_missing")
+
+ if base_schema_version is None or target_schema_version is None:
+ return _mark_unavailable(result, "serialized_dag_schema_version_missing")
+
+ unsupported_versions = [
+ version
+ for version in (base_schema_version, target_schema_version)
+ if version not in SUPPORTED_SERIALIZED_DAG_SCHEMA_VERSIONS
+ ]
+ if unsupported_versions:
+ return _mark_unavailable(
+ result, f"unsupported_serialized_dag_schema_version:{unsupported_versions[0]}"
+ )
+
+ try:
+ base_document = _canonicalize_payload_v1(base_data)
+ target_document = _canonicalize_payload_v1(target_data)
+ base_document["provenance"] = _canonicalize_value(dict(base_provenance or {}), path=("provenance",))
+ target_document["provenance"] = _canonicalize_value(
+ dict(target_provenance or {}), path=("provenance",)
+ )
+ except (AttributeError, KeyError, OverflowError, TypeError, ValueError) as error:
+ log.warning(
+ "Serialized Dag diff canonicalization failed",
+ error_type=type(error).__name__,
+ base_schema_version=base_schema_version,
+ target_schema_version=target_schema_version,
+ )
+ return _mark_unavailable(result, "serialized_dag_canonicalization_failed")
+
+ collector = _ChangeCollector(max_changes=max_changes, include_values=include_values)
+ try:
+ _collect_changes(base_document, target_document, path=(), collector=collector)
+ except _JsonEncodingError:
+ log.warning(
+ "Serialized Dag diff JSON encoding failed",
+ base_schema_version=base_schema_version,
+ target_schema_version=target_schema_version,
+ )
+ return _mark_unavailable(result, "serialized_dag_json_encoding_failed")
+
+ result["changes"] = collector.changes
+ result["truncated"] = collector.is_truncated
+ if include_values:
+ result["values"] = {"status": "available"}
+ return result
+
+
+class _ChangeCollector:
+ def __init__(self, *, max_changes: int, include_values: bool) -> None:
+ self.changes: list[dict[str, Any]] = []
+ self.count = 0
+ self.max_changes = max_changes
+ self.include_values = include_values
+
+ @property
+ def is_truncated(self) -> bool:
+ return self.count > self.max_changes
+
+ def add(
+ self,
+ *,
+ path: tuple[str, ...],
+ operation: Literal["added", "removed", "changed"],
+ before: Any,
+ after: Any,
+ ) -> None:
+ self.count += 1
+ if len(self.changes) >= self.max_changes:
+ return
+
+ public_path = _get_public_path(path)
+ category = _get_category(public_path)
+ change: dict[str, Any] = {
+ "path": _format_path(path if self.include_values else public_path),
+ "operation": operation,
+ "category": category,
+ "impact": _get_impact(category),
+ }
+ if self.include_values:
+ change["before_digest"] = None if before is _MISSING else _get_digest(before)
+ change["after_digest"] = None if after is _MISSING else _get_digest(after)
+ if before is not _MISSING:
+ change["before_value"] = before
+ if after is not _MISSING:
+ change["after_value"] = after
+ self.changes.append(change)
+
+
+_MISSING = object()
+
+
+def validate_max_changes(max_changes: int) -> None:
+ if not isinstance(max_changes, int) or isinstance(max_changes, bool) or max_changes < 1:
+ raise ValueError("max_changes must be a positive integer")
+ if max_changes > MAX_ALLOWED_CHANGES:
+ raise ValueError(f"max_changes must not exceed {MAX_ALLOWED_CHANGES}")
+
+
+def _get_schema_version(data: Mapping[str, Any] | None) -> int | None:
+ if not isinstance(data, Mapping):
+ return None
+ version = data.get("__version")
+ return version if isinstance(version, int) and not isinstance(version, bool) else None
+
+
+def _mark_unavailable(result: dict[str, Any], reason: str) -> dict[str, Any]:
+ result["mode"] = "unavailable"
+ result["unavailable_reason"] = reason
+ return result
+
+
+def _build_diff_result(base_schema_version: int | None, target_schema_version: int | None) -> dict[str, Any]:
+ return {
+ "diff_schema_version": DIFF_SCHEMA_VERSION,
+ "serialized_dag_schema_versions": {
+ "base": base_schema_version,
+ "target": target_schema_version,
+ },
+ "mode": "observed_state",
+ "changes": [],
+ "truncated": False,
+ # Always present so that reading result["values"]["status"] is safe for every caller
+ # of every entry point, whatever the outcome.
+ "values": {"status": "unavailable"},
+ }
+
+
+def _canonicalize_payload_v1(data: dict[str, Any]) -> dict[str, Any]:
+ payload = copy.deepcopy(data)
+ version = _get_schema_version(payload)
+ if version is None:
+ raise ValueError("missing or invalid __version")
+ if version == 1:
+ DagSerialization.conversion_v1_to_v2(payload)
+ DagSerialization.conversion_v2_to_v3(payload)
+ elif version == 2:
+ DagSerialization.conversion_v2_to_v3(payload)
+ if not isinstance(payload.get("dag"), Mapping):
+ raise ValueError("missing dag object")
+ dag_defaults = {
+ field: value
+ for field, value in DagSerialization.get_schema_defaults("dag").items()
+ # Dag callback flags are enabled by their presence, even when their value is false.
+ if field not in _DAG_CALLBACK_FIELDS
+ }
+ payload["dag"] = {**dag_defaults, **payload["dag"]}
+ for field in _DAG_CALLBACK_FIELDS & payload["dag"].keys():
+ payload["dag"][field] = True
+ if "params" in payload["dag"]:
+ payload["dag"]["params"] = _normalize_params(payload["dag"]["params"])
+ _apply_task_defaults(payload)
+ payload.pop("__version", None)
+ return _canonicalize_value(payload, path=())
+
+
+def _apply_task_defaults(payload: dict[str, Any]) -> None:
+ client_defaults = payload.pop("client_defaults", None)
+ if client_defaults is None:
+ client_defaults = {}
+ if not isinstance(client_defaults, Mapping):
+ raise ValueError("client_defaults is not an object")
+
+ # Fail loudly on a section this version cannot fold in: dropping it would silently
+ # compare two payloads as equal when the unhandled defaults actually differ.
+ if unknown_sections := client_defaults.keys() - {"tasks"}:
+ raise ValueError(f"unsupported client_defaults sections: {sorted(unknown_sections)}")
+
+ task_defaults = client_defaults.get("tasks", {})
+ if not isinstance(task_defaults, Mapping):
+ raise ValueError("client_defaults.tasks is not an object")
+
+ schema_defaults = DagSerialization.get_schema_defaults("operator")
+ partial_fields = (
+ SerializedBaseOperator.get_serialized_fields() - SerializedMappedOperator.get_serialized_fields()
+ )
+ # A mapped task resolves these through partial_kwargs, so their defaults have to be applied
+ # there and encoded like any other partial value rather than left at the outer level.
+ partial_schema_defaults = {
+ field: value for field, value in schema_defaults.items() if field in partial_fields
+ }
+ outer_schema_defaults = {
+ field: value for field, value in schema_defaults.items() if field not in partial_fields
+ }
+ # set_task_dag_references falls back to the Dag's dates, and the serializer elides a task date
+ # that already matches, so an absent task date means the Dag's date rather than a change.
+ inherited_dates = {field: payload["dag"].get(field) for field in ("start_date", "end_date")}
+ tasks = payload["dag"].get("tasks", [])
+ if not isinstance(tasks, list):
+ raise ValueError("dag.tasks is not a list")
+ for task in tasks:
+ if not isinstance(task, dict) or not isinstance(task.get("__var"), Mapping):
+ raise ValueError("task entry is not an object")
+ encoded_task = OperatorSerialization._apply_defaults_to_encoded_op(
+ dict(task["__var"]), dict(client_defaults)
+ )
+ upgraded_task = OperatorSerialization._upgrade_encoded_operator(encoded_task)
+ # Operator identity is selected before client defaults are applied during hydration.
+ upgraded_task["_is_mapped"] = bool(task["__var"].get("_is_mapped", False))
+ template_fields = upgraded_task.get("template_fields", [])
+ if upgraded_task.get("_is_mapped"):
+ task_data = {**outer_schema_defaults, **upgraded_task}
+ partial_kwargs = task_data.get("partial_kwargs", {})
+ if not isinstance(partial_kwargs, Mapping):
+ raise ValueError("partial_kwargs is not an object")
+ # populate_operator only folds client defaults into partial_kwargs when the payload
+ # carries the key, so an absent one leaves the top-level value as the effective value.
+ effective_partial_kwargs = (
+ {field: _encode_partial_field_value(field, value) for field, value in task_defaults.items()}
+ if "partial_kwargs" in task_data
+ else {}
+ )
+ for field, value in partial_kwargs.items():
+ if isinstance(value, Mapping) and "__type" in value and "__var" in value:
+ effective_partial_kwargs[field] = value
+ else:
+ effective_partial_kwargs[field] = _encode_partial_field_value(field, value)
+ # Match populate_operator: partial values take precedence over outer task defaults.
+ for field in partial_fields & task_data.keys():
+ value = task_data.pop(field)
+ # Outer fields are already encoded unless template handling bypasses deserialization.
+ if field in template_fields:
+ value = _encode_json_value(value)
+ effective_partial_kwargs.setdefault(field, value)
+ for field, value in partial_schema_defaults.items():
+ effective_partial_kwargs.setdefault(field, _encode_partial_field_value(field, value))
+ task_data["partial_kwargs"] = effective_partial_kwargs
+ _normalize_retry_backoff(effective_partial_kwargs)
+ else:
+ task_data = {**schema_defaults, **upgraded_task}
+ _normalize_retry_backoff(task_data)
+ for field in _OPERATOR_TIMEDELTA_FIELDS & task_data.keys():
+ value = task_data[field]
+ task_data[field] = (
+ _encode_json_value(value)
+ if field in template_fields and field in upgraded_task
+ else _encode_partial_field_value(field, value)
+ )
+ for field, dag_date in inherited_dates.items():
+ if task_data.get(field) is None:
+ task_data[field] = _encode_partial_field_value(field, dag_date)
+ elif field in template_fields:
+ task_data[field] = _encode_json_value(task_data[field])
+ else:
+ task_data[field] = _encode_partial_field_value(field, task_data[field])
+ if "params" in task_data and "params" not in template_fields:
+ task_data["params"] = _normalize_params(task_data["params"])
+ task["__var"] = task_data
+
+
+def _normalize_params(params: Any) -> Any:
+ """Normalize legacy Params without losing the order used by the trigger form."""
+ if isinstance(params, Mapping):
+ # 2.9.2 and earlier stored params as a JSON object instead of ordered pairs.
+ pairs: Any = params.items()
+ elif isinstance(params, list):
+ pairs = params
+ else:
+ return params
+
+ normalized: dict[str, Any] = {}
+ for pair in pairs:
+ if not isinstance(pair, (list, tuple)) or len(pair) != 2:
+ # Leave an unrecognised shape to the raw comparison rather than guessing at it.
+ return params
+ name, value = pair
+ if isinstance(value, Mapping) and "__class" in value:
+ value = {
+ "default": _normalize_param_attribute(value.get("default")),
+ "description": _normalize_param_attribute(value.get("description")),
+ "schema": _normalize_param_attribute(value.get("schema") or {}),
+ "source": _normalize_param_attribute(value.get("source")),
+ }
+ else:
+ value = {
+ "default": _encode_json_value(value),
+ "description": None,
+ "schema": _encode_json_value({}),
+ "source": None,
+ }
+ normalized[str(name)] = value
+ return [[name, value] for name, value in normalized.items()]
+
+
+def _normalize_param_attribute(value: Any) -> Any:
+ # Match _deserialize_param's legacy-encoding detection without hydrating user objects.
+ if isinstance(value, Mapping) and "__type" in value:
+ return value
+ if isinstance(value, list) and all(isinstance(item, Mapping) and "__type" in item for item in value):
+ return value
+ return _encode_json_value(value)
+
+
+def _encode_partial_field_value(field: str, value: Any) -> Any:
+ if field in _OPERATOR_TIMEDELTA_FIELDS and value is not None:
+ return {"__type": "timedelta", "__var": value}
+ if field.endswith("_date") and value is not None and not isinstance(value, str):
+ return {"__type": "datetime", "__var": value}
+ if field == "resources":
+ # Resources serialize as a raw mapping; a plain dictionary has a dict envelope instead.
+ return value
+ return _encode_json_value(value)
+
+
+def _encode_json_value(value: Any) -> Any:
+ """Encode plain JSON without invoking object serializers."""
+ if isinstance(value, Mapping):
+ return {"__type": "dict", "__var": {key: _encode_json_value(item) for key, item in value.items()}}
+ if isinstance(value, list):
+ return [_encode_json_value(item) for item in value]
+ return value
+
+
+def _normalize_retry_backoff(task_fields: dict[str, Any]) -> None:
+ if "retry_exponential_backoff" in task_fields:
+ value = task_fields["retry_exponential_backoff"]
+ task_fields["retry_exponential_backoff"] = 2.0 if value is True else float(value)
+
+
+def _canonicalize_value(value: Any, *, path: tuple[str, ...]) -> Any:
+ if isinstance(value, Mapping):
+ if path == ("dag", "deadline"):
+ if value.get("__type") == "deadline_alert":
+ value = value["__var"]
+ value = {"name": None, **value}
+ interval = value.get("interval")
+ if isinstance(interval, (int, float)) and not isinstance(interval, bool):
+ # Diff schema v1 uses the SDK's version-2 timedelta encoding for legacy seconds.
+ value["interval"] = {
+ "__classname__": "datetime.timedelta",
+ "__version__": 2,
+ "__data__": timedelta(seconds=interval).total_seconds(),
+ }
+ return {
+ canonical_key: _canonicalize_value(item, path=path + (canonical_key,))
+ for canonical_key, item in (
+ (_canonicalize_mapping_key(key), item)
+ for key, item in sorted(value.items(), key=lambda item: _canonicalize_mapping_key(item[0]))
+ )
+ }
+ if isinstance(value, (list, tuple)):
+ canonical_values = [_canonicalize_value(item, path=path) for item in value]
+ if path == ("dag", "tasks"):
+ return _canonicalize_keyed_list(canonical_values, _get_task_id, path)
+ if path == ("dag", "dag_dependencies"):
+ return _canonicalize_keyed_list(canonical_values, _get_dependency_key, path)
+ if path in _ORDER_INSENSITIVE_LIST_PATHS:
+ return _canonicalize_keyed_list(canonical_values, _get_string_key, path)
+ return canonical_values
+ return value
+
+
+def _canonicalize_mapping_key(key: Any) -> str:
+ if isinstance(key, str) and isinstance(key, Enum):
+ return key.value
+ return str(key)
+
+
+def _canonicalize_keyed_list(
+ values: list[Any], key_getter: Callable[[Any], str], path: tuple[str, ...]
+) -> dict[str, Any]:
+ keyed_values: dict[str, Any] = {}
+ for value in values:
+ key = key_getter(value)
+ if key in keyed_values:
+ if path in _ORDER_INSENSITIVE_LIST_PATHS or (
+ path == ("dag", "dag_dependencies") and keyed_values[key] == value
+ ):
+ continue
+ raise ValueError(f"duplicate key {key!r} in /{'/'.join(path)}")
+ keyed_values[key] = value
+ return {key: keyed_values[key] for key in sorted(keyed_values)}
+
+
+def _get_task_id(task: Any) -> str:
+ if not isinstance(task, Mapping):
+ raise ValueError("task entry is not an object")
+ task_data = task.get("__var", task)
+ task_id = task_data.get("task_id") if isinstance(task_data, Mapping) else None
+ if not isinstance(task_id, str):
+ raise ValueError("task entry has no task_id")
+ return task_id
+
+
+def _get_string_key(value: Any) -> str:
+ if not isinstance(value, str):
+ raise ValueError("collection entry is not a string")
+ return value
+
+
+def _get_dependency_key(dependency: Any) -> str:
+ if not isinstance(dependency, Mapping):
+ raise ValueError("dependency entry is not an object")
+ components = (
+ dependency.get("dependency_type"),
+ dependency.get("dependency_id"),
+ dependency.get("source"),
+ dependency.get("target"),
+ dependency.get("label"),
+ )
+ return json.dumps(components, ensure_ascii=False, separators=(",", ":"))
+
+
+def _collect_changes(
+ before: Any,
+ after: Any,
+ *,
+ path: tuple[str, ...],
+ collector: _ChangeCollector,
+) -> None:
+ if before is _MISSING and after is _MISSING:
+ return
+ if isinstance(before, Mapping) and isinstance(after, Mapping):
+ # The walk shape never depends on ``include_values``: authorization decides what each
+ # change carries, not which changes exist, so ``max_changes`` means one thing and a
+ # caller allowed values never receives a less complete change set than a redacted one.
+ if _is_task_mapping_path(path):
+ _collect_task_changes(before, after, path=path, collector=collector)
+ return
+ if _is_task_group_mapping_path(path):
+ _collect_public_field_changes(
+ before,
+ after,
+ path=path,
+ public_fields=_DIFF_V1_PUBLIC_TASK_GROUP_FIELDS,
+ collector=collector,
+ )
+ return
+ if not _should_recurse_mapping(path):
+ if not _is_json_equal(before, after):
+ collector.add(path=path, operation="changed", before=before, after=after)
+ return
+ keys = sorted({str(key) for key in before} | {str(key) for key in after})
+ for key in keys:
+ before_value = before.get(key, _MISSING)
+ after_value = after.get(key, _MISSING)
+ _collect_changes(before_value, after_value, path=path + (key,), collector=collector)
+ if collector.is_truncated:
+ return
+ return
+
+ if isinstance(before, list) and isinstance(after, list):
+ if (
+ _is_task_group_child_path(path)
+ and len(before) == len(after) == 2
+ and before[0] == after[0] == "taskgroup"
+ and isinstance(before[1], Mapping)
+ and isinstance(after[1], Mapping)
+ ):
+ _collect_changes(before[1], after[1], path=path + ("1",), collector=collector)
+ return
+ if not _is_json_equal(before, after):
+ collector.add(path=path, operation="changed", before=before, after=after)
+ return
+
+ if before is _MISSING:
+ collector.add(path=path, operation="added", before=_MISSING, after=after)
+ elif after is _MISSING:
+ collector.add(path=path, operation="removed", before=before, after=_MISSING)
+ elif not _is_json_equal(before, after):
+ collector.add(path=path, operation="changed", before=before, after=after)
+
+
+def _collect_task_changes(
+ before: Mapping[str, Any],
+ after: Mapping[str, Any],
+ *,
+ path: tuple[str, ...],
+ collector: _ChangeCollector,
+) -> None:
+ if len(path) == 3:
+ _collect_changes(
+ before.get("__type", _MISSING),
+ after.get("__type", _MISSING),
+ path=path + ("__type",),
+ collector=collector,
+ )
+ if collector.is_truncated:
+ return
+ before, after = before["__var"], after["__var"]
+ public_fields = _DIFF_V1_PUBLIC_TASK_FIELDS - {"__type"}
+ else:
+ public_fields = _DIFF_V1_PUBLIC_PARTIAL_TASK_FIELDS
+ _collect_public_field_changes(before, after, path=path, public_fields=public_fields, collector=collector)
+
+
+def _collect_public_field_changes(
+ before: Mapping[str, Any],
+ after: Mapping[str, Any],
+ *,
+ path: tuple[str, ...],
+ public_fields: frozenset[str] | set[str],
+ collector: _ChangeCollector,
+) -> None:
+ """Walk allowlisted fields one by one and report everything else as a single change."""
+ keys = {str(key) for key in before} | {str(key) for key in after}
+ for key in sorted(keys & public_fields):
+ _collect_changes(
+ before.get(key, _MISSING),
+ after.get(key, _MISSING),
+ path=path + (key,),
+ collector=collector,
+ )
+ if collector.is_truncated:
+ return
+
+ before_custom_fields = {key: before[key] for key in before if key not in public_fields}
+ after_custom_fields = {key: after[key] for key in after if key not in public_fields}
+ if not _is_json_equal(before_custom_fields, after_custom_fields):
+ collector.add(
+ path=path + (_CUSTOM_TASK_FIELDS_PATH_COMPONENT,),
+ operation="changed",
+ before=before_custom_fields,
+ after=after_custom_fields,
+ )
+
+
+def _is_task_mapping_path(path: tuple[str, ...]) -> bool:
+ return path[:2] == ("dag", "tasks") and (
+ len(path) == 3 or (len(path) == 4 and path[3] == "partial_kwargs")
+ )
+
+
+def _is_task_group_mapping_path(path: tuple[str, ...]) -> bool:
+ index = _get_task_group_field_index(path)
+ return index is not None and len(path) == index
+
+
+def _should_recurse_mapping(path: tuple[str, ...]) -> bool:
+ if path in _RECURSIVE_MAPPING_PATHS:
+ return True
+ index = _get_task_group_field_index(path)
+ return index is not None and path[index:] == ("children",)
+
+
+def _get_task_group_field_index(path: tuple[str, ...]) -> int | None:
+ if path[:2] != ("dag", "task_group"):
+ return None
+ index = 2
+ while len(path) >= index + 3 and path[index] == "children" and path[index + 2] == "1":
+ index += 3
+ return index
+
+
+def _is_task_group_child_path(path: tuple[str, ...]) -> bool:
+ index = _get_task_group_field_index(path)
+ return index is not None and len(path) == index + 2 and path[index] == "children"
+
+
+def _get_public_path(path: tuple[str, ...]) -> tuple[str, ...]:
+ if path[:2] == ("dag", "task_group"):
+ public_path = list(path)
+ index = 2
+ while len(path) >= index + 2 and path[index] == "children":
+ public_path[index + 1] = "*"
+ if len(path) < index + 3 or path[index + 2] != "1":
+ return tuple(public_path)
+ index += 3
+ if len(path) > index and path[index] not in _DIFF_V1_PUBLIC_TASK_GROUP_FIELDS:
+ public_path[index] = "custom_fields"
+ return tuple(public_path)
+ if len(path) >= 3 and path[:2] in _KEYED_COLLECTION_PATHS:
+ path = (*path[:2], "*", *path[3:])
+ if len(path) >= 4 and path[:2] == ("dag", "tasks"):
+ field_index = 4 if path[3] == "partial_kwargs" and len(path) >= 5 else 3
+ public_fields = (
+ _DIFF_V1_PUBLIC_PARTIAL_TASK_FIELDS if field_index == 4 else _DIFF_V1_PUBLIC_TASK_FIELDS
+ )
+ if path[field_index] not in public_fields:
+ return (*path[:field_index], _CUSTOM_TASK_FIELDS_PATH_COMPONENT)
+ return path
+
+
+def _format_path(path: tuple[str, ...]) -> str:
+ return "/" + "/".join(component.replace("~", "~0").replace("/", "~1") for component in path)
+
+
+class _JsonEncodingError(TypeError):
+ """Distinguish unsupported JSON values from comparison implementation errors."""
+
+
+def _serialize_canonical_json(value: Any) -> str:
+ try:
+ return json.dumps(value, ensure_ascii=False, separators=(",", ":"), sort_keys=True)
+ except TypeError as error:
+ raise _JsonEncodingError from error
+
+
+def _is_json_equal(before: Any, after: Any) -> bool:
+ # Compare canonical JSON, not Python ==, so a JSON type change (1 vs 1.0, True vs 1,
+ # False vs 0) registers as a change instead of collapsing under Python equality.
+ return _serialize_canonical_json(before) == _serialize_canonical_json(after)
+
+
+def _get_digest(value: Any) -> str:
+ return f"sha256:{hashlib.sha256(_serialize_canonical_json(value).encode()).hexdigest()}"
Review Comment:
SHA-256 provides deterministic content fingerprints here; it is not used for password storage or verification.
Digests accompany their raw values only when disclosure is authorized; redacted responses contain neither.
This appears to be a false positive for the password-hashing rule.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add serialized Dag version diff engine [airflow]
Posted by "kaxil (via GitHub)" <gi...@apache.org>.
kaxil commented on PR #69864:
URL: https://github.com/apache/airflow/pull/69864#issuecomment-5592845305
One note that doesn't fit inline: the description says the diff engine "is fully exercised by unit tests." The precedence issue I flagged in `_canonicalize_payload_v1` got through because the mapped-task default tests only shadow client defaults via `partial_kwargs`. No case sets a partial-only field at the task's *top level*, so that resolution path has no coverage at all. The one-line fix I tried passed all 186 existing tests, which is how I noticed the gap rather than the bug.
Probably worth a parametrized case that sets `retries`/`pool` at the top level with a conflicting `client_defaults.tasks` value, and softening the claim in the description.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add serialized Dag version diff engine [airflow]
Posted by "ephraimbuddy (via GitHub)" <gi...@apache.org>.
ephraimbuddy commented on code in PR #69864:
URL: https://github.com/apache/airflow/pull/69864#discussion_r4005500872
##########
airflow-core/src/airflow/serialization/dag_version_diff.py:
##########
@@ -0,0 +1,896 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""Observed-state diffs for serialized Dag payloads."""
+
+from __future__ import annotations
+
+import copy
+import hashlib
+import json
+from collections.abc import Callable, Mapping
+from datetime import timedelta
+from enum import Enum
+from typing import Any, Literal
+
+import structlog
+
+from airflow.serialization.definitions.baseoperator import SerializedBaseOperator
+from airflow.serialization.definitions.mappedoperator import SerializedMappedOperator
+from airflow.serialization.serialized_objects import (
+ _DAG_CALLBACK_FIELDS,
+ _OPERATOR_TIMEDELTA_FIELDS,
+ DagSerialization,
+ OperatorSerialization,
+)
+
+log = structlog.get_logger(__name__)
+
+DIFF_SCHEMA_VERSION = 1
+DEFAULT_MAX_CHANGES = 500
+MAX_ALLOWED_CHANGES = 5000
+SUPPORTED_SERIALIZED_DAG_SCHEMA_VERSIONS = frozenset((1, 2, 3))
+
+_ORDER_INSENSITIVE_LIST_PATHS = {
+ ("dag", "tags"),
+ ("dag", "allowed_run_types"),
+}
+_KEYED_COLLECTION_PATHS = {
+ ("dag", "tasks"),
+ ("dag", "dag_dependencies"),
+ *_ORDER_INSENSITIVE_LIST_PATHS,
+}
+_CUSTOM_TASK_FIELDS_PATH_COMPONENT = "custom_fields"
+# This allowlist is part of diff schema v1. Serializer schema changes must not
+# silently change the paths visible to callers of the diff API.
+_DIFF_V1_PUBLIC_TASK_FIELDS = frozenset(
+ {
+ "__type",
+ "_disallow_kwargs_override",
+ "_expand_input_attr",
+ "_is_mapped",
+ "_is_sensor",
+ "_logger_name",
+ "_needs_expansion",
+ "_operator_extra_links",
+ "_task_display_name",
+ "_task_module",
+ "allow_nested_operators",
+ "depends_on_past",
+ "do_xcom_push",
+ "doc",
+ "doc_json",
+ "doc_md",
+ "doc_rst",
+ "doc_yaml",
+ "downstream_task_ids",
+ "email_on_failure",
+ "email_on_retry",
+ "end_date",
+ "execution_timeout",
+ "executor",
+ "executor_config",
+ "has_on_execute_callback",
+ "has_on_failure_callback",
+ "has_on_retry_callback",
+ "has_on_skipped_callback",
+ "has_on_success_callback",
+ "ignore_first_depends_on_past",
+ "inlets",
+ "is_setup",
+ "is_teardown",
+ "map_index_template",
+ "max_active_tis_per_dag",
+ "max_active_tis_per_dagrun",
+ "max_retry_delay",
+ "multiple_outputs",
+ "on_failure_fail_dagrun",
+ "outlets",
+ "owner",
+ "params",
+ "partial_kwargs",
+ "pool",
+ "pool_slots",
+ "priority_weight",
+ "queue",
+ "render_template_as_native_obj",
+ "retries",
+ "retry_delay",
+ "retry_exponential_backoff",
+ "start_date",
+ "start_from_trigger",
+ "start_trigger_args",
+ "task_id",
+ "task_type",
+ "template_ext",
+ "template_fields",
+ "template_fields_renderers",
+ "trigger_rule",
+ "ui_color",
+ "ui_fgcolor",
+ "wait_for_downstream",
+ "wait_for_past_depends_before_skipping",
+ "weight_rule",
+ }
+)
+_DIFF_V1_REDACTED_SCHEMA_TASK_FIELDS = frozenset({"_arg_bindings"})
+# _get_category classifies task fields with these; every name must be a public task field or the
+# entry is unreachable, since a non-public field is aggregated under custom_fields before lookup.
+_DIFF_V1_TASK_ASSET_FIELDS = frozenset({"inlets", "outlets"})
+_DIFF_V1_TASK_PARAM_FIELDS = frozenset({"params"})
+_DIFF_V1_TASK_DEPENDENCY_FIELDS = frozenset({"downstream_task_ids"})
+_DIFF_V1_TASK_METADATA_FIELDS = frozenset(
+ {
+ "doc",
+ "doc_json",
+ "doc_md",
+ "doc_rst",
+ "doc_yaml",
+ "owner",
+ "ui_color",
+ "ui_fgcolor",
+ "_task_display_name",
+ "task_display_name",
+ }
+)
+_DIFF_V1_PUBLIC_PARTIAL_TASK_FIELDS = _DIFF_V1_PUBLIC_TASK_FIELDS | {"task_display_name"}
+# Classify every Dag schema field explicitly so new fields require a policy decision.
+_DIFF_V1_DAG_FIELD_CATEGORIES = {
+ "_concurrency": "schedule",
+ "_processor_dags_folder": "provenance",
+ "access_control": "authorization",
+ "allowed_run_types": "schedule",
+ "bundle_name": "provenance",
+ "catchup": "schedule",
+ "dag_dependencies": "dependency",
+ "dag_display_name": "metadata",
+ "dag_id": "metadata",
+ "dagrun_timeout": "schedule",
+ "deadline": "deadline",
+ "default_args": "param",
+ "description": "metadata",
+ "disable_bundle_versioning": "task",
+ "doc_md": "metadata",
+ "edge_info": "metadata",
+ "end_date": "schedule",
+ "fail_fast": "schedule",
+ "fileloc": "provenance",
+ "has_on_failure_callback": "callback",
+ "has_on_success_callback": "callback",
+ "is_paused_upon_creation": "schedule",
+ "max_active_runs": "schedule",
+ "max_active_tasks": "schedule",
+ "max_consecutive_failed_dag_runs": "schedule",
+ "owner_links": "metadata",
+ "params": "param",
+ "relative_fileloc": "provenance",
+ "render_template_as_native_obj": "task",
+ "rerun_with_latest_version": "task",
+ "start_date": "schedule",
+ "tags": "metadata",
+ "task_group": "task",
+ "tasks": "task",
+ "timetable": "schedule",
+ "timezone": "schedule",
+}
+_DIFF_V1_LEGACY_DAG_FIELD_CATEGORIES = {
+ "fail_stop": "schedule",
+ "on_failure_callback": "callback",
+ "on_success_callback": "callback",
+ "schedule": "schedule",
+ "schedule_interval": "schedule",
+}
+_RECURSIVE_MAPPING_PATHS = {
+ (),
+ ("dag",),
+ ("provenance",),
+ *_KEYED_COLLECTION_PATHS,
+}
+_DIFF_V1_TASK_GROUP_METADATA_FIELDS = frozenset(
+ {"group_display_name", "tooltip", "doc_md", "ui_color", "ui_fgcolor"}
+)
+_DIFF_V1_PUBLIC_TASK_GROUP_FIELDS = _DIFF_V1_TASK_GROUP_METADATA_FIELDS | {
+ "_group_id",
+ "prefix_group_id",
+ "children",
+ "upstream_group_ids",
+ "downstream_group_ids",
+ "upstream_task_ids",
+ "downstream_task_ids",
+ "expand_input",
+ "is_mapped",
+}
+
+
+def build_unavailable_dag_diff(
+ *,
+ base_data: dict[str, Any] | None,
+ target_data: dict[str, Any] | None,
+ reason: str,
+) -> dict[str, Any]:
+ """Report a known unavailable reason without comparing the stored payloads."""
+ return _mark_unavailable(
+ _build_diff_result(_get_schema_version(base_data), _get_schema_version(target_data)), reason
+ )
+
+
+def build_serialized_dag_diff(
+ *,
+ base_data: dict[str, Any] | None,
+ target_data: dict[str, Any] | None,
+ base_provenance: Mapping[str, Any] | None = None,
+ target_provenance: Mapping[str, Any] | None = None,
+ include_values: bool = False,
+ max_changes: int = DEFAULT_MAX_CHANGES,
+) -> dict[str, Any]:
+ """
+ Build a bounded, deterministic diff from two stored serialized Dag payloads.
+
+ Raw values, digests, and value-derived path components are returned only when
+ ``include_values`` is true. Callers must authorize disclosure of the entire
+ serialized payload, including access-control role names and permission mappings,
+ before enabling it.
+ """
+ validate_max_changes(max_changes)
+
+ base_schema_version = _get_schema_version(base_data)
+ target_schema_version = _get_schema_version(target_data)
+ result = _build_diff_result(base_schema_version, target_schema_version)
+
+ if base_data is None or target_data is None:
+ return _mark_unavailable(result, "serialized_dag_missing")
+
+ if base_schema_version is None or target_schema_version is None:
+ return _mark_unavailable(result, "serialized_dag_schema_version_missing")
+
+ unsupported_versions = [
+ version
+ for version in (base_schema_version, target_schema_version)
+ if version not in SUPPORTED_SERIALIZED_DAG_SCHEMA_VERSIONS
+ ]
+ if unsupported_versions:
+ return _mark_unavailable(
+ result, f"unsupported_serialized_dag_schema_version:{unsupported_versions[0]}"
+ )
+
+ try:
+ base_document = _canonicalize_payload_v1(base_data)
+ target_document = _canonicalize_payload_v1(target_data)
+ base_document["provenance"] = _canonicalize_value(dict(base_provenance or {}), path=("provenance",))
+ target_document["provenance"] = _canonicalize_value(
+ dict(target_provenance or {}), path=("provenance",)
+ )
+ except (AttributeError, KeyError, OverflowError, TypeError, ValueError) as error:
+ log.warning(
+ "Serialized Dag diff canonicalization failed",
+ error_type=type(error).__name__,
+ base_schema_version=base_schema_version,
+ target_schema_version=target_schema_version,
+ )
+ return _mark_unavailable(result, "serialized_dag_canonicalization_failed")
+
+ collector = _ChangeCollector(max_changes=max_changes, include_values=include_values)
+ try:
+ _collect_changes(base_document, target_document, path=(), collector=collector)
+ except _JsonEncodingError:
+ log.warning(
+ "Serialized Dag diff JSON encoding failed",
+ base_schema_version=base_schema_version,
+ target_schema_version=target_schema_version,
+ )
+ return _mark_unavailable(result, "serialized_dag_json_encoding_failed")
+
+ result["changes"] = collector.changes
+ result["truncated"] = collector.is_truncated
+ if include_values:
+ result["values"] = {"status": "available"}
+ return result
+
+
+class _ChangeCollector:
+ def __init__(self, *, max_changes: int, include_values: bool) -> None:
+ self.changes: list[dict[str, Any]] = []
+ self.count = 0
+ self.max_changes = max_changes
+ self.include_values = include_values
+
+ @property
+ def is_truncated(self) -> bool:
+ return self.count > self.max_changes
+
+ def add(
+ self,
+ *,
+ path: tuple[str, ...],
+ operation: Literal["added", "removed", "changed"],
+ before: Any,
+ after: Any,
+ ) -> None:
+ self.count += 1
+ if len(self.changes) >= self.max_changes:
+ return
+
+ public_path = _get_public_path(path)
+ category = _get_category(public_path)
+ change: dict[str, Any] = {
+ "path": _format_path(path if self.include_values else public_path),
+ "operation": operation,
+ "category": category,
+ "impact": _get_impact(category),
+ }
+ if self.include_values:
+ change["before_digest"] = None if before is _MISSING else _get_digest(before)
+ change["after_digest"] = None if after is _MISSING else _get_digest(after)
+ if before is not _MISSING:
+ change["before_value"] = before
+ if after is not _MISSING:
+ change["after_value"] = after
+ self.changes.append(change)
+
+
+_MISSING = object()
+
+
+def validate_max_changes(max_changes: int) -> None:
+ if not isinstance(max_changes, int) or isinstance(max_changes, bool) or max_changes < 1:
+ raise ValueError("max_changes must be a positive integer")
+ if max_changes > MAX_ALLOWED_CHANGES:
+ raise ValueError(f"max_changes must not exceed {MAX_ALLOWED_CHANGES}")
+
+
+def _get_schema_version(data: Mapping[str, Any] | None) -> int | None:
+ if not isinstance(data, Mapping):
+ return None
+ version = data.get("__version")
+ return version if isinstance(version, int) and not isinstance(version, bool) else None
+
+
+def _mark_unavailable(result: dict[str, Any], reason: str) -> dict[str, Any]:
+ result["mode"] = "unavailable"
+ result["unavailable_reason"] = reason
+ return result
+
+
+def _build_diff_result(base_schema_version: int | None, target_schema_version: int | None) -> dict[str, Any]:
+ return {
+ "diff_schema_version": DIFF_SCHEMA_VERSION,
+ "serialized_dag_schema_versions": {
+ "base": base_schema_version,
+ "target": target_schema_version,
+ },
+ "mode": "observed_state",
+ "changes": [],
+ "truncated": False,
+ # Always present so that reading result["values"]["status"] is safe for every caller
+ # of every entry point, whatever the outcome.
+ "values": {"status": "unavailable"},
+ }
+
+
+def _canonicalize_payload_v1(data: dict[str, Any]) -> dict[str, Any]:
+ payload = copy.deepcopy(data)
+ version = _get_schema_version(payload)
+ if version is None:
+ raise ValueError("missing or invalid __version")
+ if version == 1:
+ DagSerialization.conversion_v1_to_v2(payload)
+ DagSerialization.conversion_v2_to_v3(payload)
+ elif version == 2:
+ DagSerialization.conversion_v2_to_v3(payload)
+ if not isinstance(payload.get("dag"), Mapping):
+ raise ValueError("missing dag object")
+ dag_defaults = {
+ field: value
+ for field, value in DagSerialization.get_schema_defaults("dag").items()
+ # Dag callback flags are enabled by their presence, even when their value is false.
+ if field not in _DAG_CALLBACK_FIELDS
+ }
+ payload["dag"] = {**dag_defaults, **payload["dag"]}
+ for field in _DAG_CALLBACK_FIELDS & payload["dag"].keys():
+ payload["dag"][field] = True
+ if "params" in payload["dag"]:
+ payload["dag"]["params"] = _normalize_params(payload["dag"]["params"])
+ _apply_task_defaults(payload)
+ payload.pop("__version", None)
+ return _canonicalize_value(payload, path=())
+
+
+def _apply_task_defaults(payload: dict[str, Any]) -> None:
+ client_defaults = payload.pop("client_defaults", None)
+ if client_defaults is None:
+ client_defaults = {}
+ if not isinstance(client_defaults, Mapping):
+ raise ValueError("client_defaults is not an object")
+
+ # Fail loudly on a section this version cannot fold in: dropping it would silently
+ # compare two payloads as equal when the unhandled defaults actually differ.
+ if unknown_sections := client_defaults.keys() - {"tasks"}:
+ raise ValueError(f"unsupported client_defaults sections: {sorted(unknown_sections)}")
+
+ task_defaults = client_defaults.get("tasks", {})
+ if not isinstance(task_defaults, Mapping):
+ raise ValueError("client_defaults.tasks is not an object")
+
+ schema_defaults = DagSerialization.get_schema_defaults("operator")
+ partial_fields = (
+ SerializedBaseOperator.get_serialized_fields() - SerializedMappedOperator.get_serialized_fields()
+ )
+ # A mapped task resolves these through partial_kwargs, so their defaults have to be applied
+ # there and encoded like any other partial value rather than left at the outer level.
+ partial_schema_defaults = {
+ field: value for field, value in schema_defaults.items() if field in partial_fields
+ }
+ outer_schema_defaults = {
+ field: value for field, value in schema_defaults.items() if field not in partial_fields
+ }
+ # set_task_dag_references falls back to the Dag's dates, and the serializer elides a task date
+ # that already matches, so an absent task date means the Dag's date rather than a change.
+ inherited_dates = {field: payload["dag"].get(field) for field in ("start_date", "end_date")}
+ tasks = payload["dag"].get("tasks", [])
+ if not isinstance(tasks, list):
+ raise ValueError("dag.tasks is not a list")
+ for task in tasks:
+ if not isinstance(task, dict) or not isinstance(task.get("__var"), Mapping):
+ raise ValueError("task entry is not an object")
+ encoded_task = OperatorSerialization._apply_defaults_to_encoded_op(
+ dict(task["__var"]), dict(client_defaults)
+ )
+ upgraded_task = OperatorSerialization._upgrade_encoded_operator(encoded_task)
+ # Operator identity is selected before client defaults are applied during hydration.
+ upgraded_task["_is_mapped"] = bool(task["__var"].get("_is_mapped", False))
+ template_fields = upgraded_task.get("template_fields", [])
+ if upgraded_task.get("_is_mapped"):
+ task_data = {**outer_schema_defaults, **upgraded_task}
+ partial_kwargs = task_data.get("partial_kwargs", {})
+ if not isinstance(partial_kwargs, Mapping):
+ raise ValueError("partial_kwargs is not an object")
+ # populate_operator only folds client defaults into partial_kwargs when the payload
+ # carries the key, so an absent one leaves the top-level value as the effective value.
+ effective_partial_kwargs = (
+ {field: _encode_partial_field_value(field, value) for field, value in task_defaults.items()}
+ if "partial_kwargs" in task_data
+ else {}
+ )
+ for field, value in partial_kwargs.items():
+ if isinstance(value, Mapping) and "__type" in value and "__var" in value:
+ effective_partial_kwargs[field] = value
+ else:
+ effective_partial_kwargs[field] = _encode_partial_field_value(field, value)
+ # Match populate_operator: partial values take precedence over outer task defaults.
+ for field in partial_fields & task_data.keys():
+ value = task_data.pop(field)
+ # Outer fields are already encoded unless template handling bypasses deserialization.
+ if field in template_fields:
+ value = _encode_json_value(value)
+ effective_partial_kwargs.setdefault(field, value)
+ for field, value in partial_schema_defaults.items():
+ effective_partial_kwargs.setdefault(field, _encode_partial_field_value(field, value))
+ task_data["partial_kwargs"] = effective_partial_kwargs
+ _normalize_retry_backoff(effective_partial_kwargs)
+ else:
+ task_data = {**schema_defaults, **upgraded_task}
+ _normalize_retry_backoff(task_data)
+ for field in _OPERATOR_TIMEDELTA_FIELDS & task_data.keys():
+ value = task_data[field]
+ task_data[field] = (
+ _encode_json_value(value)
+ if field in template_fields and field in upgraded_task
+ else _encode_partial_field_value(field, value)
+ )
+ for field, dag_date in inherited_dates.items():
+ if task_data.get(field) is None:
+ task_data[field] = _encode_partial_field_value(field, dag_date)
+ elif field in template_fields:
+ task_data[field] = _encode_json_value(task_data[field])
+ else:
+ task_data[field] = _encode_partial_field_value(field, task_data[field])
+ if "params" in task_data and "params" not in template_fields:
+ task_data["params"] = _normalize_params(task_data["params"])
+ task["__var"] = task_data
+
+
+def _normalize_params(params: Any) -> Any:
+ """Normalize legacy Params without losing the order used by the trigger form."""
+ if isinstance(params, Mapping):
+ # 2.9.2 and earlier stored params as a JSON object instead of ordered pairs.
+ pairs: Any = params.items()
+ elif isinstance(params, list):
+ pairs = params
+ else:
+ return params
+
+ normalized: dict[str, Any] = {}
+ for pair in pairs:
+ if not isinstance(pair, (list, tuple)) or len(pair) != 2:
+ # Leave an unrecognised shape to the raw comparison rather than guessing at it.
+ return params
+ name, value = pair
+ if isinstance(value, Mapping) and "__class" in value:
+ value = {
+ "default": _normalize_param_attribute(value.get("default")),
+ "description": _normalize_param_attribute(value.get("description")),
+ "schema": _normalize_param_attribute(value.get("schema") or {}),
+ "source": _normalize_param_attribute(value.get("source")),
+ }
+ else:
+ value = {
+ "default": _encode_json_value(value),
+ "description": None,
+ "schema": _encode_json_value({}),
+ "source": None,
+ }
+ normalized[str(name)] = value
+ return [[name, value] for name, value in normalized.items()]
+
+
+def _normalize_param_attribute(value: Any) -> Any:
+ # Match _deserialize_param's legacy-encoding detection without hydrating user objects.
+ if isinstance(value, Mapping) and "__type" in value:
+ return value
+ if isinstance(value, list) and all(isinstance(item, Mapping) and "__type" in item for item in value):
+ return value
+ return _encode_json_value(value)
+
+
+def _encode_partial_field_value(field: str, value: Any) -> Any:
+ if field in _OPERATOR_TIMEDELTA_FIELDS and value is not None:
+ return {"__type": "timedelta", "__var": value}
+ if field.endswith("_date") and value is not None and not isinstance(value, str):
+ return {"__type": "datetime", "__var": value}
+ if field == "resources":
+ # Resources serialize as a raw mapping; a plain dictionary has a dict envelope instead.
+ return value
+ return _encode_json_value(value)
+
+
+def _encode_json_value(value: Any) -> Any:
+ """Encode plain JSON without invoking object serializers."""
+ if isinstance(value, Mapping):
+ return {"__type": "dict", "__var": {key: _encode_json_value(item) for key, item in value.items()}}
+ if isinstance(value, list):
+ return [_encode_json_value(item) for item in value]
+ return value
+
+
+def _normalize_retry_backoff(task_fields: dict[str, Any]) -> None:
+ if "retry_exponential_backoff" in task_fields:
+ value = task_fields["retry_exponential_backoff"]
+ task_fields["retry_exponential_backoff"] = 2.0 if value is True else float(value)
+
+
+def _canonicalize_value(value: Any, *, path: tuple[str, ...]) -> Any:
+ if isinstance(value, Mapping):
+ if path == ("dag", "deadline"):
+ if value.get("__type") == "deadline_alert":
+ value = value["__var"]
+ value = {"name": None, **value}
+ interval = value.get("interval")
+ if isinstance(interval, (int, float)) and not isinstance(interval, bool):
+ # Diff schema v1 uses the SDK's version-2 timedelta encoding for legacy seconds.
+ value["interval"] = {
+ "__classname__": "datetime.timedelta",
+ "__version__": 2,
+ "__data__": timedelta(seconds=interval).total_seconds(),
+ }
+ return {
+ canonical_key: _canonicalize_value(item, path=path + (canonical_key,))
+ for canonical_key, item in (
+ (_canonicalize_mapping_key(key), item)
+ for key, item in sorted(value.items(), key=lambda item: _canonicalize_mapping_key(item[0]))
+ )
+ }
+ if isinstance(value, (list, tuple)):
+ canonical_values = [_canonicalize_value(item, path=path) for item in value]
+ if path == ("dag", "tasks"):
+ return _canonicalize_keyed_list(canonical_values, _get_task_id, path)
+ if path == ("dag", "dag_dependencies"):
+ return _canonicalize_keyed_list(canonical_values, _get_dependency_key, path)
+ if path in _ORDER_INSENSITIVE_LIST_PATHS:
+ return _canonicalize_keyed_list(canonical_values, _get_string_key, path)
+ return canonical_values
+ return value
+
+
+def _canonicalize_mapping_key(key: Any) -> str:
+ if isinstance(key, str) and isinstance(key, Enum):
+ return key.value
+ return str(key)
+
+
+def _canonicalize_keyed_list(
+ values: list[Any], key_getter: Callable[[Any], str], path: tuple[str, ...]
+) -> dict[str, Any]:
+ keyed_values: dict[str, Any] = {}
+ for value in values:
+ key = key_getter(value)
+ if key in keyed_values:
+ if path in _ORDER_INSENSITIVE_LIST_PATHS or (
+ path == ("dag", "dag_dependencies") and keyed_values[key] == value
+ ):
+ continue
+ raise ValueError(f"duplicate key {key!r} in /{'/'.join(path)}")
+ keyed_values[key] = value
+ return {key: keyed_values[key] for key in sorted(keyed_values)}
+
+
+def _get_task_id(task: Any) -> str:
+ if not isinstance(task, Mapping):
+ raise ValueError("task entry is not an object")
+ task_data = task.get("__var", task)
+ task_id = task_data.get("task_id") if isinstance(task_data, Mapping) else None
+ if not isinstance(task_id, str):
+ raise ValueError("task entry has no task_id")
+ return task_id
+
+
+def _get_string_key(value: Any) -> str:
+ if not isinstance(value, str):
+ raise ValueError("collection entry is not a string")
+ return value
+
+
+def _get_dependency_key(dependency: Any) -> str:
+ if not isinstance(dependency, Mapping):
+ raise ValueError("dependency entry is not an object")
+ components = (
+ dependency.get("dependency_type"),
+ dependency.get("dependency_id"),
+ dependency.get("source"),
+ dependency.get("target"),
+ dependency.get("label"),
+ )
+ return json.dumps(components, ensure_ascii=False, separators=(",", ":"))
+
+
+def _collect_changes(
+ before: Any,
+ after: Any,
+ *,
+ path: tuple[str, ...],
+ collector: _ChangeCollector,
+) -> None:
+ if before is _MISSING and after is _MISSING:
+ return
+ if isinstance(before, Mapping) and isinstance(after, Mapping):
+ # The walk shape never depends on ``include_values``: authorization decides what each
+ # change carries, not which changes exist, so ``max_changes`` means one thing and a
+ # caller allowed values never receives a less complete change set than a redacted one.
+ if _is_task_mapping_path(path):
+ _collect_task_changes(before, after, path=path, collector=collector)
+ return
+ if _is_task_group_mapping_path(path):
+ _collect_public_field_changes(
+ before,
+ after,
+ path=path,
+ public_fields=_DIFF_V1_PUBLIC_TASK_GROUP_FIELDS,
+ collector=collector,
+ )
+ return
+ if not _should_recurse_mapping(path):
+ if not _is_json_equal(before, after):
+ collector.add(path=path, operation="changed", before=before, after=after)
+ return
+ keys = sorted({str(key) for key in before} | {str(key) for key in after})
+ for key in keys:
+ before_value = before.get(key, _MISSING)
+ after_value = after.get(key, _MISSING)
+ _collect_changes(before_value, after_value, path=path + (key,), collector=collector)
+ if collector.is_truncated:
+ return
+ return
+
+ if isinstance(before, list) and isinstance(after, list):
+ if (
+ _is_task_group_child_path(path)
+ and len(before) == len(after) == 2
+ and before[0] == after[0] == "taskgroup"
+ and isinstance(before[1], Mapping)
+ and isinstance(after[1], Mapping)
+ ):
+ _collect_changes(before[1], after[1], path=path + ("1",), collector=collector)
+ return
+ if not _is_json_equal(before, after):
+ collector.add(path=path, operation="changed", before=before, after=after)
+ return
+
+ if before is _MISSING:
+ collector.add(path=path, operation="added", before=_MISSING, after=after)
+ elif after is _MISSING:
+ collector.add(path=path, operation="removed", before=before, after=_MISSING)
+ elif not _is_json_equal(before, after):
+ collector.add(path=path, operation="changed", before=before, after=after)
+
+
+def _collect_task_changes(
+ before: Mapping[str, Any],
+ after: Mapping[str, Any],
+ *,
+ path: tuple[str, ...],
+ collector: _ChangeCollector,
+) -> None:
+ if len(path) == 3:
+ _collect_changes(
+ before.get("__type", _MISSING),
+ after.get("__type", _MISSING),
+ path=path + ("__type",),
+ collector=collector,
+ )
+ if collector.is_truncated:
+ return
+ before, after = before["__var"], after["__var"]
+ public_fields = _DIFF_V1_PUBLIC_TASK_FIELDS - {"__type"}
+ else:
+ public_fields = _DIFF_V1_PUBLIC_PARTIAL_TASK_FIELDS
+ _collect_public_field_changes(before, after, path=path, public_fields=public_fields, collector=collector)
+
+
+def _collect_public_field_changes(
+ before: Mapping[str, Any],
+ after: Mapping[str, Any],
+ *,
+ path: tuple[str, ...],
+ public_fields: frozenset[str] | set[str],
+ collector: _ChangeCollector,
+) -> None:
+ """Walk allowlisted fields one by one and report everything else as a single change."""
+ keys = {str(key) for key in before} | {str(key) for key in after}
+ for key in sorted(keys & public_fields):
+ _collect_changes(
+ before.get(key, _MISSING),
+ after.get(key, _MISSING),
+ path=path + (key,),
+ collector=collector,
+ )
+ if collector.is_truncated:
+ return
+
+ before_custom_fields = {key: before[key] for key in before if key not in public_fields}
+ after_custom_fields = {key: after[key] for key in after if key not in public_fields}
+ if not _is_json_equal(before_custom_fields, after_custom_fields):
+ collector.add(
+ path=path + (_CUSTOM_TASK_FIELDS_PATH_COMPONENT,),
+ operation="changed",
+ before=before_custom_fields,
+ after=after_custom_fields,
+ )
+
+
+def _is_task_mapping_path(path: tuple[str, ...]) -> bool:
+ return path[:2] == ("dag", "tasks") and (
+ len(path) == 3 or (len(path) == 4 and path[3] == "partial_kwargs")
+ )
+
+
+def _is_task_group_mapping_path(path: tuple[str, ...]) -> bool:
+ index = _get_task_group_field_index(path)
+ return index is not None and len(path) == index
+
+
+def _should_recurse_mapping(path: tuple[str, ...]) -> bool:
+ if path in _RECURSIVE_MAPPING_PATHS:
+ return True
+ index = _get_task_group_field_index(path)
+ return index is not None and path[index:] == ("children",)
+
+
+def _get_task_group_field_index(path: tuple[str, ...]) -> int | None:
+ if path[:2] != ("dag", "task_group"):
+ return None
+ index = 2
+ while len(path) >= index + 3 and path[index] == "children" and path[index + 2] == "1":
+ index += 3
+ return index
+
+
+def _is_task_group_child_path(path: tuple[str, ...]) -> bool:
+ index = _get_task_group_field_index(path)
+ return index is not None and len(path) == index + 2 and path[index] == "children"
+
+
+def _get_public_path(path: tuple[str, ...]) -> tuple[str, ...]:
+ if path[:2] == ("dag", "task_group"):
+ public_path = list(path)
+ index = 2
+ while len(path) >= index + 2 and path[index] == "children":
+ public_path[index + 1] = "*"
+ if len(path) < index + 3 or path[index + 2] != "1":
+ return tuple(public_path)
+ index += 3
+ if len(path) > index and path[index] not in _DIFF_V1_PUBLIC_TASK_GROUP_FIELDS:
+ public_path[index] = "custom_fields"
+ return tuple(public_path)
+ if len(path) >= 3 and path[:2] in _KEYED_COLLECTION_PATHS:
+ path = (*path[:2], "*", *path[3:])
+ if len(path) >= 4 and path[:2] == ("dag", "tasks"):
+ field_index = 4 if path[3] == "partial_kwargs" and len(path) >= 5 else 3
+ public_fields = (
+ _DIFF_V1_PUBLIC_PARTIAL_TASK_FIELDS if field_index == 4 else _DIFF_V1_PUBLIC_TASK_FIELDS
+ )
+ if path[field_index] not in public_fields:
+ return (*path[:field_index], _CUSTOM_TASK_FIELDS_PATH_COMPONENT)
+ return path
+
+
+def _format_path(path: tuple[str, ...]) -> str:
+ return "/" + "/".join(component.replace("~", "~0").replace("/", "~1") for component in path)
+
+
+class _JsonEncodingError(TypeError):
+ """Distinguish unsupported JSON values from comparison implementation errors."""
+
+
+def _serialize_canonical_json(value: Any) -> str:
+ try:
+ return json.dumps(value, ensure_ascii=False, separators=(",", ":"), sort_keys=True)
+ except TypeError as error:
+ raise _JsonEncodingError from error
+
+
+def _is_json_equal(before: Any, after: Any) -> bool:
+ # Compare canonical JSON, not Python ==, so a JSON type change (1 vs 1.0, True vs 1,
+ # False vs 0) registers as a change instead of collapsing under Python equality.
+ return _serialize_canonical_json(before) == _serialize_canonical_json(after)
+
+
+def _get_digest(value: Any) -> str:
+ return f"sha256:{hashlib.sha256(_serialize_canonical_json(value).encode()).hexdigest()}"
Review Comment:
SHA-256 here fingerprints canonical serialized diff content; this code is not used for password storage or verification. CodeQL traces a value from `Connection.to_dict()["password"]`, but hashing content that may contain a password does not make this a password-protection mechanism.
`_ChangeCollector.add()` only produces digests when `include_values=True`, and each digest is returned alongside its corresponding raw value. `DagVersion.get_diff()` defaults to `values_status="unavailable"`; callers must authorize disclosure of the full serialized payload before enabling values. Redacted responses omit both values and digests, which existing regression tests verify.
The digest therefore reveals no additional value to the recipient beyond the raw content already authorized for disclosure. A password KDF would not improve protection here; SHA-256 serves deterministic content fingerprinting. This is a false positive for `py/weak-sensitive-data-hashing`.
---
Drafted-by: Codex (GPT-6) (no human review before posting)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add serialized Dag version diff engine [airflow]
Posted by "github-advanced-security[bot] (via GitHub)" <gi...@apache.org>.
github-advanced-security[bot] commented on code in PR #69864:
URL: https://github.com/apache/airflow/pull/69864#discussion_r4017165412
##########
airflow-core/src/airflow/serialization/dag_version_diff.py:
##########
@@ -0,0 +1,1222 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""
+Observed-state diffs for serialized Dag payloads.
+
+Diff schema v1 -- the wire format every entry point here returns.
+:func:`build_serialized_dag_diff` and :func:`build_unavailable_dag_diff` both return one
+dictionary in the shape described below, and ``DIFF_SCHEMA_VERSION`` is the version of that
+shape: bump it whenever a client-observable part of this contract changes.
+
+Top-level keys, all always present:
+
+* ``diff_schema_version`` -- ``int``, the value of ``DIFF_SCHEMA_VERSION``.
+* ``serialized_dag_schema_versions`` -- ``{"base": int | None, "target": int | None}``, the
+ ``__version`` each stored payload carried. ``None`` means the key was absent or not an
+ integer; ``bool`` is rejected, so ``__version: true`` reads as ``None`` and never as
+ version 1. A version in ``SUPPORTED_SERIALIZED_DAG_SCHEMA_VERSIONS`` is upgraded to the
+ newest supported version before anything is compared.
+* ``mode`` -- ``"observed_state"`` when the two payloads were compared, ``"unavailable"``
+ when no comparison happened.
+* ``changes`` -- the change records described below, ordered by the deterministic walk.
+ Always empty when ``mode`` is ``"unavailable"``.
+* ``truncated`` -- ``True`` when more underlying changes exist than ``max_changes`` allowed.
+ Always ``False`` when ``mode`` is ``"unavailable"``.
+* ``values`` -- ``{"status": "available" | "unavailable"}``. ``"available"`` only for an
+ ``"observed_state"`` result whose caller authorized value disclosure, so
+ ``result["values"]["status"]`` is safe to read on every result of every entry point.
+
+``unavailable_reason`` is the one conditional key: present exactly when ``mode`` is
+``"unavailable"``. :func:`build_serialized_dag_diff` produces:
+
+* ``serialized_dag_missing`` -- one side has no stored payload.
+* ``serialized_dag_schema_version_missing`` -- one side carries no usable ``__version``.
+* ``unsupported_serialized_dag_schema_version:<n>`` -- ``<n>`` is the first unsupported
+ version found, base before target.
+* ``serialized_dag_canonicalization_failed`` -- a payload could not be normalized (malformed
+ structure, an unsupported ``client_defaults`` section, an unencodable value).
+* ``serialized_dag_recursion_limit_exceeded`` -- the comparison walk ran too deep.
+* ``serialized_dag_json_encoding_failed`` -- a value could not be encoded as canonical JSON.
+
+:meth:`~airflow.models.dag_version.DagVersion.get_diff` reuses
+``serialized_dag_decode_failed``, for a stored payload that will not decompress or parse, and
+``deadline_alert_missing``, for a payload that references a deadline alert row that is gone.
+:func:`build_unavailable_dag_diff` passes its caller's reason through unchanged, so a new
+caller extends this list rather than inventing an undocumented value.
+
+Each change record carries these keys in both disclosure modes:
+
+* ``path`` -- a JSON-Pointer-style path into a synthetic document with two roots: ``/dag``
+ (the canonicalized serialized Dag, with ``__version`` dropped and client defaults folded
+ into the tasks) and ``/provenance`` (the provenance mapping the caller supplied --
+ ``bundle_name``, ``bundle_version`` and ``version_data`` from ``get_diff``). ``~`` and
+ ``/`` inside a component are escaped as ``~0`` and ``~1``.
+* ``operation`` -- ``"added"``, ``"removed"`` or ``"changed"``.
+* ``category`` -- a :data:`DiffCategory`: ``task`` for task definitions and the Dag fields
+ that shape task execution, ``dependency`` for ``dag_dependencies`` and downstream task
+ ids, ``schedule`` for timetable, dates and concurrency, ``param`` for params,
+ ``asset`` for task inlets and outlets, ``deadline`` for deadline alerts,
+ ``callback`` for callback presence, ``metadata`` for descriptive and display fields,
+ ``authorization`` for ``access_control``, ``provenance`` for everything under
+ ``/provenance`` plus the file and bundle locators, and ``unknown`` for anything
+ unclassified. ``_DIFF_V1_DAG_FIELD_CATEGORIES`` is the authoritative Dag-field mapping;
+ an unclassified task field falls back to ``task``, which over-reports impact.
+ ``default_args`` is the one Dag field classified per key rather than as a whole, because its
+ keys are the operator fields they will be applied to: an allowlisted key takes that field's
+ category (``owner`` is ``metadata``, ``retries`` is ``task``), and the aggregate record for
+ the rest keeps the conservative ``task``. Only a stored value that is not a readable dict
+ envelope classifies as ``task`` as a whole.
+* ``impact`` -- a :data:`DiffImpact` derived from ``category``: ``provenance``, ``metadata``
+ and ``authorization`` carry through unchanged, every operational category becomes
+ ``execution``, and ``unknown`` stays ``unknown``.
+* ``occurrence_count`` -- how many underlying changes the record stands for.
+
+A caller that authorized values (``include_values=True``, ``values.status`` ``"available"``)
+additionally gets:
+
+* ``before_digest`` / ``after_digest`` -- ``"sha256:<hex>"`` over the canonical JSON of that
+ side, or ``None`` when that side is missing. Both keys are always present.
+* ``before_value`` / ``after_value`` -- the canonicalized value, which is the stored one after
+ schema and client defaults are folded in and params are normalized. ``before_value`` is absent from
+ an ``added`` record and ``after_value`` from a ``removed`` one, which is how a missing side
+ is told apart from a stored ``null``.
+
+``path`` and ``occurrence_count`` also differ between the modes. A redacted record reports the
+public path: members of keyed collections (``/dag/tasks``, ``/dag/dag_dependencies``,
+``/dag/tags``, ``/dag/allowed_run_types`` and task group children) collapse to ``*`` instead
+of naming a task, tag or group, and records sharing a public path and operation are merged
+into one whose ``occurrence_count`` counts the merged changes. An authorized record keeps the
+identifying component and always has an ``occurrence_count`` of 1. Fields outside the v1 task,
+task group and ``default_args`` allowlists are aggregated into a single ``custom_fields`` record
+in both modes, so neither a private serializer field nor a user-chosen ``default_args`` key is
+ever named by either -- ``/dag/default_args`` names only allowlisted operator field names, which
+is why its keys can be classified at all. A top-level key outside the v1 root allowlist is
+likewise reported as ``/custom_fields`` when redacted, while an authorized record keeps its real
+root path.
+
+``max_changes`` bounds how many underlying changes either mode admits, so authorization decides
+what a change carries and never which changes exist. Admission is what the bound limits, not
+the record count: a redacted diff can stop admitting while holding fewer records than the
+bound, because several admitted changes merged into one record. A change reaching a group
+already in the result is counted without consuming a new record, so repeats keep accumulating
+until a change that needs a new record stops the walk.
+
+``truncated`` reports that such a change was dropped. When it is ``False`` every changed path
+is present with an exact ``occurrence_count``. When it is ``True`` the walk stopped early: some
+paths are absent rather than unchanged, and every ``occurrence_count`` is a lower bound, since
+occurrences after the stop were never reached.
+
+Typed dictionaries for the result and the change record are deliberately deferred -- the
+record shape varies with disclosure mode, so modelling it belongs with the REST layer that
+exposes it. :data:`DiffCategory` and :data:`DiffImpact` are stable enough to generate enums
+from today.
+"""
+
+from __future__ import annotations
+
+import copy
+import hashlib
+import json
+from collections.abc import Callable, Mapping, MutableMapping
+from contextlib import suppress
+from dataclasses import dataclass
+from datetime import timedelta
+from enum import Enum
+from typing import Any, Literal, NamedTuple
+
+import structlog
+
+from airflow.serialization.definitions.baseoperator import SerializedBaseOperator
+from airflow.serialization.definitions.mappedoperator import SerializedMappedOperator
+from airflow.serialization.serialized_objects import (
+ _DAG_CALLBACK_FIELDS,
+ _OPERATOR_TIMEDELTA_FIELDS,
+ DagSerialization,
+ OperatorSerialization,
+)
+
+log = structlog.get_logger(__name__)
+
+DIFF_SCHEMA_VERSION = 1
+DEFAULT_MAX_CHANGES = 500
+MAX_ALLOWED_CHANGES = 5000
+SUPPORTED_SERIALIZED_DAG_SCHEMA_VERSIONS = frozenset((1, 2, 3))
+
+DiffCategory = Literal[
+ "asset",
+ "authorization",
+ "callback",
+ "deadline",
+ "dependency",
+ "metadata",
+ "param",
+ "provenance",
+ "schedule",
+ "task",
+ "unknown",
+]
+DiffImpact = Literal["authorization", "execution", "metadata", "provenance", "unknown"]
+
+_ORDER_INSENSITIVE_LIST_PATHS = {
+ ("dag", "tags"),
+ ("dag", "allowed_run_types"),
+}
+_KEYED_COLLECTION_PATHS = {
+ ("dag", "tasks"),
+ ("dag", "dag_dependencies"),
+ *_ORDER_INSENSITIVE_LIST_PATHS,
+}
+# Canonicalization folds away __version and client_defaults and adds provenance, so these are the
+# only top-level sections the walk is allowed to name.
+_DIFF_V1_PUBLIC_ROOT_FIELDS = frozenset({"dag", "provenance"})
+_CUSTOM_TASK_FIELDS_PATH_COMPONENT = "custom_fields"
+# This allowlist is part of diff schema v1. Serializer schema changes must not
+# silently change the paths visible to callers of the diff API.
+_DIFF_V1_PUBLIC_TASK_FIELDS = frozenset(
+ {
+ "__type",
+ "_can_skip_downstream",
+ "_disallow_kwargs_override",
+ "_expand_input_attr",
+ "_is_empty",
+ "_is_mapped",
+ "_is_sensor",
+ "_logger_name",
+ "_needs_expansion",
+ "_operator_extra_links",
+ "_operator_name",
+ "_task_display_name",
+ "_task_module",
+ "allow_nested_operators",
+ "depends_on_past",
+ "do_xcom_push",
+ "doc",
+ "doc_json",
+ "doc_md",
+ "doc_rst",
+ "doc_yaml",
+ "downstream_task_ids",
+ "email",
+ "email_on_failure",
+ "email_on_retry",
+ "end_date",
+ "execution_timeout",
+ "executor",
+ "executor_config",
+ "expand_input",
+ "has_on_execute_callback",
+ "has_on_failure_callback",
+ "has_on_retry_callback",
+ "has_on_skipped_callback",
+ "has_on_success_callback",
+ "has_retry_policy",
+ "ignore_first_depends_on_past",
+ "inlets",
+ "is_setup",
+ "is_stub",
+ "is_teardown",
+ "map_index_template",
+ "max_active_tis_per_dag",
+ "max_active_tis_per_dagrun",
+ "max_retry_delay",
+ "multiple_outputs",
+ "on_failure_fail_dagrun",
+ "op_kwargs_expand_input",
+ "outlets",
+ "owner",
+ "params",
+ "partial_kwargs",
+ "pool",
+ "pool_slots",
+ "priority_weight",
+ "python_callable_name",
+ "queue",
+ "render_template_as_native_obj",
+ "reschedule",
+ "resources",
+ "retries",
+ "retry_delay",
+ "retry_exponential_backoff",
+ "run_as_user",
+ "start_date",
+ "start_from_trigger",
+ "start_trigger_args",
+ "task_id",
+ "task_type",
+ "template_ext",
+ "template_fields",
+ "template_fields_renderers",
+ "trigger_rule",
+ "ui_color",
+ "ui_fgcolor",
+ "wait_for_downstream",
+ "wait_for_past_depends_before_skipping",
+ "weight_rule",
+ }
+)
+_DIFF_V1_REDACTED_SCHEMA_TASK_FIELDS = frozenset({"_arg_bindings"})
+# Task-field categories used by _get_category.
+_DIFF_V1_TASK_ASSET_FIELDS = frozenset({"inlets", "outlets"})
+_DIFF_V1_TASK_PARAM_FIELDS = frozenset({"params"})
+_DIFF_V1_TASK_DEPENDENCY_FIELDS = frozenset({"downstream_task_ids"})
+_DIFF_V1_TASK_METADATA_FIELDS = frozenset(
+ {
+ "_operator_name",
+ "doc",
+ "doc_json",
+ "doc_md",
+ "doc_rst",
+ "doc_yaml",
+ "owner",
+ "ui_color",
+ "ui_fgcolor",
+ "_task_display_name",
+ "task_display_name",
+ }
+)
+_DIFF_V1_PUBLIC_PARTIAL_TASK_FIELDS = _DIFF_V1_PUBLIC_TASK_FIELDS | {"task_display_name"}
+_DEFAULT_ARGS_PATH = ("dag", "default_args")
+# A Dag's default_args holds the same operator __init__ kwargs partial_kwargs holds -- the
+# serializer rewrites _HAS_FLAG_FIELDS in both -- so it reuses that allowlist. "__type" names the
+# stored value's own dict envelope rather than a key inside it.
+_DIFF_V1_PUBLIC_DEFAULT_ARGS_FIELDS = _DIFF_V1_PUBLIC_PARTIAL_TASK_FIELDS - {"__type"}
+# Classify every Dag schema field explicitly so new fields require a policy decision.
+_DIFF_V1_DAG_FIELD_CATEGORIES: dict[str, DiffCategory] = {
+ "_concurrency": "schedule",
+ "_processor_dags_folder": "provenance",
+ "access_control": "authorization",
+ "allowed_run_types": "schedule",
+ "bundle_name": "provenance",
+ "catchup": "schedule",
+ "dag_dependencies": "dependency",
+ "dag_display_name": "metadata",
+ "dag_id": "metadata",
+ "dagrun_timeout": "schedule",
+ "deadline": "deadline",
+ # Only reached when the stored value is not the dict envelope _collect_default_args_changes
+ # walks per key; an unreadable envelope keeps the conservative impact of its widest key.
+ "default_args": "task",
+ "description": "metadata",
+ # Decides whether a run pins a bundle version, so it changes which code later runs execute
+ # and whether triggering a historical version is allowed at all.
+ "disable_bundle_versioning": "task",
+ "doc_md": "metadata",
+ "edge_info": "metadata",
+ "end_date": "schedule",
+ "fail_fast": "schedule",
+ "fileloc": "provenance",
+ "has_on_failure_callback": "callback",
+ "has_on_success_callback": "callback",
+ "is_paused_upon_creation": "schedule",
+ "max_active_runs": "schedule",
+ "max_active_tasks": "schedule",
+ "max_consecutive_failed_dag_runs": "schedule",
+ "owner_links": "metadata",
+ "params": "param",
+ "relative_fileloc": "provenance",
+ "render_template_as_native_obj": "task",
+ "rerun_with_latest_version": "task",
+ "start_date": "schedule",
+ "tags": "metadata",
+ "task_group": "task",
+ "tasks": "task",
+ "timetable": "schedule",
+ "timezone": "schedule",
+}
+# Fields dropped from the current schema that a stored payload can still carry. Every entry must
+# survive _canonicalize_payload_v1, or it classifies nothing.
+_DIFF_V1_LEGACY_DAG_FIELD_CATEGORIES: dict[str, DiffCategory] = {
+ "fail_stop": "schedule",
+ "on_failure_callback": "callback",
+ "on_success_callback": "callback",
+ "schedule": "schedule",
+}
+_RECURSIVE_MAPPING_PATHS = {
+ (),
+ ("dag",),
+ ("provenance",),
+ *_KEYED_COLLECTION_PATHS,
+}
+_DIFF_V1_TASK_GROUP_METADATA_FIELDS = frozenset(
+ {"group_display_name", "tooltip", "doc_md", "ui_color", "ui_fgcolor"}
+)
+_DIFF_V1_PUBLIC_TASK_GROUP_FIELDS = _DIFF_V1_TASK_GROUP_METADATA_FIELDS | {
+ "_group_id",
+ "prefix_group_id",
+ "children",
+ "upstream_group_ids",
+ "downstream_group_ids",
+ "upstream_task_ids",
+ "downstream_task_ids",
+ "expand_input",
+ "is_mapped",
+}
+
+
+def build_unavailable_dag_diff(
+ *,
+ base_data: dict[str, Any] | None,
+ target_data: dict[str, Any] | None,
+ reason: str,
+) -> dict[str, Any]:
+ """Report a known unavailable reason without comparing the stored payloads."""
+ return _mark_unavailable(
+ _build_diff_result(_get_schema_version(base_data), _get_schema_version(target_data)), reason
+ )
+
+
+def build_serialized_dag_diff(
+ *,
+ base_data: dict[str, Any] | None,
+ target_data: dict[str, Any] | None,
+ base_provenance: Mapping[str, Any] | None = None,
+ target_provenance: Mapping[str, Any] | None = None,
+ include_values: bool = False,
+ max_changes: int = DEFAULT_MAX_CHANGES,
+) -> dict[str, Any]:
+ """
+ Compare two serialized Dag payloads and their provenance deterministically.
+
+ Callers must authorize disclosure of the entire serialized payload, including
+ access-control roles and permissions, before setting ``include_values=True``.
+ This exposes canonicalized values, digests, and identifying path components.
+
+ ``max_changes`` limits underlying changes admitted to the result. Redacted changes
+ with the same public path and operation share a record. Repeats can be counted past
+ the limit until a change needing a new record stops the walk. When ``truncated`` is
+ true, some paths are absent and occurrence counts are lower bounds.
+
+ An ``unavailable`` result includes the reason comparison failed.
+ """
+ validate_max_changes(max_changes)
+
+ base_schema_version = _get_schema_version(base_data)
+ target_schema_version = _get_schema_version(target_data)
+ result = _build_diff_result(base_schema_version, target_schema_version)
+
+ if base_data is None or target_data is None:
+ return _mark_unavailable(result, "serialized_dag_missing")
+
+ if base_schema_version is None or target_schema_version is None:
+ return _mark_unavailable(result, "serialized_dag_schema_version_missing")
+
+ unsupported_versions = [
+ version
+ for version in (base_schema_version, target_schema_version)
+ if version not in SUPPORTED_SERIALIZED_DAG_SCHEMA_VERSIONS
+ ]
+ if unsupported_versions:
+ return _mark_unavailable(
+ result, f"unsupported_serialized_dag_schema_version:{unsupported_versions[0]}"
+ )
+
+ try:
+ base_document = _canonicalize_payload_v1(base_data)
+ target_document = _canonicalize_payload_v1(target_data)
+ base_document["provenance"] = _canonicalize_value(dict(base_provenance or {}), path=("provenance",))
+ target_document["provenance"] = _canonicalize_value(
+ dict(target_provenance or {}), path=("provenance",)
+ )
+ except (AttributeError, KeyError, OverflowError, RecursionError, TypeError, ValueError) as error:
+ log.warning(
+ "Serialized Dag diff canonicalization failed",
+ error_type=type(error).__name__,
+ base_schema_version=base_schema_version,
+ target_schema_version=target_schema_version,
+ )
+ return _mark_unavailable(result, "serialized_dag_canonicalization_failed")
+
+ collector = _ChangeCollector(max_changes=max_changes, include_values=include_values)
+ try:
+ _collect_changes(base_document, target_document, path=(), collector=collector)
+ except RecursionError:
+ log.warning(
+ "Serialized Dag diff recursion limit exceeded",
+ base_schema_version=base_schema_version,
+ target_schema_version=target_schema_version,
+ )
+ return _mark_unavailable(result, "serialized_dag_recursion_limit_exceeded")
+ except _JsonEncodingError:
+ log.warning(
+ "Serialized Dag diff JSON encoding failed",
+ base_schema_version=base_schema_version,
+ target_schema_version=target_schema_version,
+ )
+ return _mark_unavailable(result, "serialized_dag_json_encoding_failed")
+
+ result["changes"] = collector.changes
+ result["truncated"] = collector.is_truncated
+ if include_values:
+ result["values"] = {"status": "available"}
+ return result
+
+
+class _ChangeCollector:
+ def __init__(self, *, max_changes: int, include_values: bool) -> None:
+ self.changes: list[dict[str, Any]] = []
+ self.count = 0
+ self.max_changes = max_changes
+ self.include_values = include_values
+ self._truncated = False
+ self._redacted_changes: dict[tuple[tuple[str, ...], str], dict[str, Any]] = {}
+
+ @property
+ def is_truncated(self) -> bool:
+ return self._truncated
+
+ def add(
+ self,
+ *,
+ path: tuple[str, ...],
+ operation: Literal["added", "removed", "changed"],
+ before: Any,
+ after: Any,
+ ) -> None:
+ self.count += 1
+ public_path = _get_public_path(path)
+ key = (public_path, operation)
+ if not self.include_values and (existing := self._redacted_changes.get(key)) is not None:
+ # Counting a repeat of an already disclosed path adds no record and reveals no new
+ # path, so it stays exact past the bound instead of silently undercounting.
+ existing["occurrence_count"] += 1
+ return
+ if self.count > self.max_changes:
+ self._truncated = True
+ return
+
+ category = _get_category(public_path)
+ change: dict[str, Any] = {
+ "path": _format_path(path if self.include_values else public_path),
+ "operation": operation,
+ "category": category,
+ "impact": _get_impact(category),
+ "occurrence_count": 1,
+ }
+ if self.include_values:
+ change["before_digest"] = None if before is _MISSING else _get_digest(before)
+ change["after_digest"] = None if after is _MISSING else _get_digest(after)
+ if before is not _MISSING:
+ change["before_value"] = before
+ if after is not _MISSING:
+ change["after_value"] = after
+ else:
+ self._redacted_changes[key] = change
+ self.changes.append(change)
+
+
+_MISSING = object()
+
+
+def validate_max_changes(max_changes: int) -> None:
+ if not isinstance(max_changes, int) or isinstance(max_changes, bool) or max_changes < 1:
+ raise ValueError("max_changes must be a positive integer")
+ if max_changes > MAX_ALLOWED_CHANGES:
+ raise ValueError(f"max_changes must not exceed {MAX_ALLOWED_CHANGES}")
+
+
+def _get_schema_version(data: Mapping[str, Any] | None) -> int | None:
+ if not isinstance(data, Mapping):
+ return None
+ version = data.get("__version")
+ return version if isinstance(version, int) and not isinstance(version, bool) else None
+
+
+def _mark_unavailable(result: dict[str, Any], reason: str) -> dict[str, Any]:
+ result["mode"] = "unavailable"
+ result["unavailable_reason"] = reason
+ return result
+
+
+def _build_diff_result(base_schema_version: int | None, target_schema_version: int | None) -> dict[str, Any]:
+ return {
+ "diff_schema_version": DIFF_SCHEMA_VERSION,
+ "serialized_dag_schema_versions": {
+ "base": base_schema_version,
+ "target": target_schema_version,
+ },
+ "mode": "observed_state",
+ "changes": [],
+ "truncated": False,
+ # Always present so reading result["values"]["status"] is safe on every outcome.
+ "values": {"status": "unavailable"},
+ }
+
+
+def _canonicalize_payload_v1(data: dict[str, Any]) -> dict[str, Any]:
+ payload = copy.deepcopy(data)
+ version = _get_schema_version(payload)
+ if version is None:
+ raise ValueError("missing or invalid __version")
+ if version == 1:
+ DagSerialization.conversion_v1_to_v2(payload)
+ DagSerialization.conversion_v2_to_v3(payload)
+ elif version == 2:
+ DagSerialization.conversion_v2_to_v3(payload)
+ if not isinstance(payload.get("dag"), Mapping):
+ raise ValueError("missing dag object")
+ dag_defaults = {
+ field: value
+ for field, value in DagSerialization.get_schema_defaults("dag").items()
+ # Dag callback flags are enabled by their presence, even when their value is false.
+ if field not in _DAG_CALLBACK_FIELDS
+ }
+ payload["dag"] = {**dag_defaults, **payload["dag"]}
+ for field in _DAG_CALLBACK_FIELDS & payload["dag"].keys():
+ payload["dag"][field] = True
+ if "params" in payload["dag"]:
+ payload["dag"]["params"] = _normalize_params(payload["dag"]["params"])
+ _apply_task_defaults(payload)
+ payload.pop("__version", None)
+ return _canonicalize_value(payload, path=())
+
+
+@dataclass(frozen=True)
+class _TaskDefaultSets:
+ """The default sets that every task in one payload is folded against."""
+
+ schema_defaults: Mapping[str, Any]
+ outer_schema_defaults: Mapping[str, Any]
+ partial_schema_defaults: Mapping[str, Any]
+ partial_fields: frozenset[str]
+ client_task_defaults: Mapping[str, Any]
+ inherited_dates: Mapping[str, Any]
+
+
+def _build_task_default_sets(
+ payload: dict[str, Any], client_task_defaults: Mapping[str, Any]
+) -> _TaskDefaultSets:
+ # get_schema_defaults is lru_cache'd and hands back the very mapping the live Dag-hydration path
+ # reads, so every use of it in this module must stay read-only. Mutating it in place would
+ # corrupt Dag hydration process-wide, not merely produce a wrong diff.
+ schema_defaults = DagSerialization.get_schema_defaults("operator")
+ partial_fields = (
+ SerializedBaseOperator.get_serialized_fields() - SerializedMappedOperator.get_serialized_fields()
+ )
+ return _TaskDefaultSets(
+ schema_defaults=schema_defaults,
+ outer_schema_defaults={
+ field: value for field, value in schema_defaults.items() if field not in partial_fields
+ },
+ # A mapped task resolves these through partial_kwargs, so their defaults have to be applied
+ # there and encoded like any other partial value rather than left at the outer level.
+ partial_schema_defaults={
+ field: value for field, value in schema_defaults.items() if field in partial_fields
+ },
+ partial_fields=partial_fields,
+ client_task_defaults=client_task_defaults,
+ # set_task_dag_references falls back to the Dag's dates, and the serializer elides a task date
+ # that already matches, so an absent task date means the Dag's date rather than a change.
+ inherited_dates={field: payload["dag"].get(field) for field in ("start_date", "end_date")},
+ )
+
+
+def _apply_task_defaults(payload: dict[str, Any]) -> None:
+ client_defaults = payload.pop("client_defaults", None)
+ if client_defaults is None:
+ client_defaults = {}
+ if not isinstance(client_defaults, Mapping):
+ raise ValueError("client_defaults is not an object")
+
+ # Fail loudly on a section this version cannot fold in: dropping it would silently
+ # compare two payloads as equal when the unhandled defaults actually differ.
+ if unknown_sections := client_defaults.keys() - {"tasks"}:
+ raise ValueError(f"unsupported client_defaults sections: {sorted(unknown_sections)}")
+
+ client_task_defaults = client_defaults.get("tasks", {})
+ if not isinstance(client_task_defaults, Mapping):
+ raise ValueError("client_defaults.tasks is not an object")
+
+ defaults = _build_task_default_sets(payload, client_task_defaults)
+ tasks = payload["dag"].get("tasks", [])
+ if not isinstance(tasks, list):
+ raise ValueError("dag.tasks is not a list")
+ for task in tasks:
+ upgraded_task = _upgrade_task_entry(task, client_defaults)
+ template_fields = upgraded_task.get("template_fields", [])
+ if upgraded_task.get("_is_mapped"):
+ task_data = _build_mapped_task_data(upgraded_task, template_fields, defaults)
+ else:
+ task_data = _build_unmapped_task_data(upgraded_task, template_fields, defaults)
+ _apply_inherited_dates(task_data, template_fields, defaults)
+ if "params" in task_data and "params" not in template_fields:
+ task_data["params"] = _normalize_params(task_data["params"])
+ task["__var"] = task_data
+
+
+def _upgrade_task_entry(task: Any, client_defaults: Mapping[str, Any]) -> dict[str, Any]:
+ if not isinstance(task, dict) or not isinstance(task.get("__var"), Mapping):
+ raise ValueError("task entry is not an object")
+ encoded_task = OperatorSerialization._apply_defaults_to_encoded_op(
+ dict(task["__var"]), dict(client_defaults)
+ )
+ upgraded_task = OperatorSerialization._upgrade_encoded_operator(encoded_task)
+ # Operator identity is selected before client defaults are applied during hydration.
+ upgraded_task["_is_mapped"] = bool(task["__var"].get("_is_mapped", False))
+ return upgraded_task
+
+
+def _build_mapped_task_data(
+ upgraded_task: dict[str, Any], template_fields: Any, defaults: _TaskDefaultSets
+) -> dict[str, Any]:
+ task_data = {**defaults.outer_schema_defaults, **upgraded_task}
+ effective_partial_kwargs = _build_effective_partial_kwargs(task_data, template_fields, defaults)
+ task_data["partial_kwargs"] = effective_partial_kwargs
+ _normalize_retry_backoff(effective_partial_kwargs)
+ return task_data
+
+
+def _build_effective_partial_kwargs(
+ task_data: dict[str, Any], template_fields: Any, defaults: _TaskDefaultSets
+) -> dict[str, Any]:
+ """Resolve a mapped task's partial_kwargs, consuming the outer fields that move into them."""
+ partial_kwargs = task_data.get("partial_kwargs", {})
+ if not isinstance(partial_kwargs, Mapping):
+ raise ValueError("partial_kwargs is not an object")
+ # populate_operator only folds client defaults into partial_kwargs when the payload
+ # carries the key, so an absent one leaves the top-level value as the effective value.
+ effective_partial_kwargs = (
+ {
+ field: _encode_partial_field_value(field, value)
+ for field, value in defaults.client_task_defaults.items()
+ }
+ if "partial_kwargs" in task_data
+ else {}
+ )
+ for field, value in partial_kwargs.items():
+ effective_partial_kwargs[field] = (
+ value if _is_encoded_partial_value(value) else _encode_partial_field_value(field, value)
+ )
+ # Match populate_operator: partial values take precedence over outer task defaults.
+ for field in defaults.partial_fields & task_data.keys():
+ value = task_data.pop(field)
+ # Outer fields are already encoded unless template handling bypasses deserialization.
+ if field in template_fields:
+ value = _encode_json_value(value)
+ effective_partial_kwargs.setdefault(field, value)
+ for field, value in defaults.partial_schema_defaults.items():
+ effective_partial_kwargs.setdefault(field, _encode_partial_field_value(field, value))
+ return effective_partial_kwargs
+
+
+def _build_unmapped_task_data(
+ upgraded_task: dict[str, Any], template_fields: Any, defaults: _TaskDefaultSets
+) -> dict[str, Any]:
+ task_data = {**defaults.schema_defaults, **upgraded_task}
+ _normalize_retry_backoff(task_data)
+ for field in _OPERATOR_TIMEDELTA_FIELDS & task_data.keys():
+ value = task_data[field]
+ task_data[field] = (
+ _encode_json_value(value)
+ if field in template_fields and field in upgraded_task
+ else _encode_partial_field_value(field, value)
+ )
+ return task_data
+
+
+def _apply_inherited_dates(
+ task_data: dict[str, Any], template_fields: Any, defaults: _TaskDefaultSets
+) -> None:
+ for field, dag_date in defaults.inherited_dates.items():
+ if task_data.get(field) is None:
+ task_data[field] = _encode_partial_field_value(field, dag_date)
+ elif field in template_fields:
+ task_data[field] = _encode_json_value(task_data[field])
+ else:
+ task_data[field] = _encode_partial_field_value(field, task_data[field])
+
+
+# These two predicates deliberately disagree on what counts as already encoded, because the
+# serializer sites they mirror disagree: keep them apart rather than "aligning" them.
+def _is_encoded_partial_value(value: Any) -> bool:
+ # _deserialize_partial_kwargs only takes the full-deserialization path with BOTH keys present.
+ return isinstance(value, Mapping) and "__type" in value and "__var" in value
+
+
+def _is_encoded_param_attribute(value: Any) -> bool:
+ # _deserialize_param detects a legacy encoding from "__type" alone, with no "__var" required.
+ if isinstance(value, Mapping):
+ return "__type" in value
+ if isinstance(value, list):
+ return all(isinstance(item, Mapping) and "__type" in item for item in value)
+ return False
+
+
+def _normalize_params(params: Any) -> Any:
+ """Normalize legacy Params without losing the order used by the trigger form."""
+ if isinstance(params, Mapping):
+ # 2.9.2 and earlier stored params as a JSON object instead of ordered pairs.
+ pairs: Any = params.items()
+ elif isinstance(params, list):
+ pairs = params
+ else:
+ return params
+
+ normalized: dict[str, Any] = {}
+ for pair in pairs:
+ if not isinstance(pair, (list, tuple)) or len(pair) != 2:
+ # Leave an unrecognised shape to the raw comparison rather than guessing at it.
+ return params
+ name, value = pair
+ if isinstance(value, Mapping) and "__class" in value:
+ value = {
+ "default": _normalize_param_attribute(value.get("default")),
+ "description": _normalize_param_attribute(value.get("description")),
+ "schema": _normalize_param_attribute(value.get("schema") or {}),
+ "source": _normalize_param_attribute(value.get("source")),
+ }
+ else:
+ value = {
+ "default": _encode_json_value(value),
+ "description": None,
+ "schema": _encode_json_value({}),
+ "source": None,
+ }
+ normalized[str(name)] = value
+ return [[name, value] for name, value in normalized.items()]
+
+
+def _normalize_param_attribute(value: Any) -> Any:
+ # Match _deserialize_param's legacy-encoding detection without hydrating user objects.
+ if _is_encoded_param_attribute(value):
+ return value
+ return _encode_json_value(value)
+
+
+def _encode_partial_field_value(field: str, value: Any) -> Any:
+ if field in _OPERATOR_TIMEDELTA_FIELDS and value is not None:
+ return {"__type": "timedelta", "__var": value}
+ if field.endswith("_date") and value is not None and not isinstance(value, str):
+ return {"__type": "datetime", "__var": value}
+ if field == "resources":
+ # Resources serialize as a raw mapping; a plain dictionary has a dict envelope instead.
+ return value
+ return _encode_json_value(value)
+
+
+def _encode_json_value(value: Any) -> Any:
+ """Encode plain JSON without invoking object serializers."""
+ if isinstance(value, Mapping):
+ return {"__type": "dict", "__var": {key: _encode_json_value(item) for key, item in value.items()}}
+ if isinstance(value, list):
+ return [_encode_json_value(item) for item in value]
+ return value
+
+
+def _normalize_retry_backoff(task_fields: dict[str, Any]) -> None:
+ if "retry_exponential_backoff" in task_fields:
+ value = task_fields["retry_exponential_backoff"]
+ # A non-numeric stored value stays as it is so the comparison reports just this field
+ # instead of collapsing the whole diff to a canonicalization failure.
+ with suppress(TypeError, ValueError):
+ task_fields["retry_exponential_backoff"] = 2.0 if value is True else float(value)
+
+
+def _canonicalize_value(value: Any, *, path: tuple[str, ...]) -> Any:
+ if isinstance(value, Mapping):
+ if path == ("dag", "deadline"):
+ if value.get("__type") == "deadline_alert":
+ value = value["__var"]
+ value = {"name": None, **value}
+ interval = value.get("interval")
+ if isinstance(interval, (int, float)) and not isinstance(interval, bool):
+ # Diff schema v1 uses the SDK's version-2 timedelta encoding for legacy seconds.
+ value["interval"] = {
+ "__classname__": "datetime.timedelta",
+ "__version__": 2,
+ "__data__": timedelta(seconds=interval).total_seconds(),
+ }
+ return {
+ canonical_key: _canonicalize_value(item, path=path + (canonical_key,))
+ for canonical_key, item in (
+ (_canonicalize_mapping_key(key), item)
+ for key, item in sorted(value.items(), key=lambda item: _canonicalize_mapping_key(item[0]))
+ )
+ }
+ if isinstance(value, (list, tuple)):
+ canonical_values = [_canonicalize_value(item, path=path) for item in value]
+ if path == ("dag", "tasks"):
+ return _canonicalize_keyed_list(canonical_values, _get_task_id, path)
+ if path == ("dag", "dag_dependencies"):
+ return _canonicalize_keyed_list(canonical_values, _get_dependency_key, path)
+ if path in _ORDER_INSENSITIVE_LIST_PATHS:
+ return _canonicalize_keyed_list(canonical_values, _get_string_key, path)
+ return canonical_values
+ return value
+
+
+def _canonicalize_mapping_key(key: Any) -> str:
+ if isinstance(key, str) and isinstance(key, Enum):
+ return key.value
+ return str(key)
+
+
+def _canonicalize_keyed_list(
+ values: list[Any], key_getter: Callable[[Any], str], path: tuple[str, ...]
+) -> dict[str, Any]:
+ keyed_values: dict[str, Any] = {}
+ for value in values:
+ key = key_getter(value)
+ if key in keyed_values:
+ if path in _ORDER_INSENSITIVE_LIST_PATHS or (
+ path == ("dag", "dag_dependencies") and keyed_values[key] == value
+ ):
+ continue
+ raise ValueError(f"duplicate key {key!r} in /{'/'.join(path)}")
+ keyed_values[key] = value
+ return {key: keyed_values[key] for key in sorted(keyed_values)}
+
+
+def _get_task_id(task: Any) -> str:
+ if not isinstance(task, Mapping):
+ raise ValueError("task entry is not an object")
+ task_data = task.get("__var", task)
+ task_id = task_data.get("task_id") if isinstance(task_data, Mapping) else None
+ if not isinstance(task_id, str):
+ raise ValueError("task entry has no task_id")
+ return task_id
+
+
+def _get_string_key(value: Any) -> str:
+ if not isinstance(value, str):
+ raise ValueError("collection entry is not a string")
+ return value
+
+
+def _get_dependency_key(dependency: Any) -> str:
+ if not isinstance(dependency, MutableMapping):
+ raise ValueError("dependency entry is not an object")
+ # DagDependency.node_id identifies an edge by the components below alone, and ``label`` only
+ # mirrors state compared elsewhere (a task's _task_display_name, or an asset name already inside
+ # dependency_id). Keying on it turns a rename into one edge removed and another added; leaving
+ # it in the entry turns two entries on one edge into duplicates that fail canonicalization.
+ dependency.pop("label", None)
+ components = (
+ dependency.get("dependency_type"),
+ dependency.get("dependency_id"),
+ dependency.get("source"),
+ dependency.get("target"),
+ )
+ return json.dumps(components, ensure_ascii=False, separators=(",", ":"))
+
+
+def _collect_changes(
+ before: Any,
+ after: Any,
+ *,
+ path: tuple[str, ...],
+ collector: _ChangeCollector,
+) -> None:
+ if before is _MISSING and after is _MISSING:
+ return
+ if isinstance(before, Mapping) and isinstance(after, Mapping):
+ # Never branch on ``include_values`` here; see build_serialized_dag_diff.
+ if _is_task_mapping_path(path):
+ _collect_task_changes(before, after, path=path, collector=collector)
+ return
+ if _is_task_group_mapping_path(path):
+ _collect_public_field_changes(
+ before,
+ after,
+ path=path,
+ public_fields=_DIFF_V1_PUBLIC_TASK_GROUP_FIELDS,
+ collector=collector,
+ )
+ return
+ if path == _DEFAULT_ARGS_PATH:
+ _collect_default_args_changes(before, after, path=path, collector=collector)
+ return
+ if not _should_recurse_mapping(path):
+ if not _is_json_equal(before, after):
+ collector.add(path=path, operation="changed", before=before, after=after)
+ return
+ keys = sorted({str(key) for key in before} | {str(key) for key in after})
+ for key in keys:
+ before_value = before.get(key, _MISSING)
+ after_value = after.get(key, _MISSING)
+ _collect_changes(before_value, after_value, path=path + (key,), collector=collector)
+ if collector.is_truncated:
+ return
+ return
+
+ if isinstance(before, list) and isinstance(after, list):
+ if (
+ _is_task_group_child_path(path)
+ and len(before) == len(after) == 2
+ and before[0] == after[0] == "taskgroup"
+ and isinstance(before[1], Mapping)
+ and isinstance(after[1], Mapping)
+ ):
+ _collect_changes(before[1], after[1], path=path + ("1",), collector=collector)
+ return
+ if not _is_json_equal(before, after):
+ collector.add(path=path, operation="changed", before=before, after=after)
+ return
+
+ if before is _MISSING:
+ collector.add(path=path, operation="added", before=_MISSING, after=after)
+ elif after is _MISSING:
+ collector.add(path=path, operation="removed", before=before, after=_MISSING)
+ elif not _is_json_equal(before, after):
+ collector.add(path=path, operation="changed", before=before, after=after)
+
+
+def _collect_task_changes(
+ before: Mapping[str, Any],
+ after: Mapping[str, Any],
+ *,
+ path: tuple[str, ...],
+ collector: _ChangeCollector,
+) -> None:
+ if len(path) == 3:
+ _collect_changes(
+ before.get("__type", _MISSING),
+ after.get("__type", _MISSING),
+ path=path + ("__type",),
+ collector=collector,
+ )
+ if collector.is_truncated:
+ return
+ before, after = before["__var"], after["__var"]
+ public_fields = _DIFF_V1_PUBLIC_TASK_FIELDS - {"__type"}
+ else:
+ public_fields = _DIFF_V1_PUBLIC_PARTIAL_TASK_FIELDS
+ _collect_public_field_changes(before, after, path=path, public_fields=public_fields, collector=collector)
+
+
+def _collect_default_args_changes(
+ before: Mapping[str, Any],
+ after: Mapping[str, Any],
+ *,
+ path: tuple[str, ...],
+ collector: _ChangeCollector,
+) -> None:
+ """Compare a Dag's default_args key by key so each key carries its own impact."""
+ if not (_is_encoded_default_args(before) and _is_encoded_default_args(after)):
+ # An envelope this version cannot read stays one opaque leaf: the allowlist was written for
+ # the keys of a dict envelope, so applying it to another shape could name something else.
+ if not _is_json_equal(before, after):
+ collector.add(path=path, operation="changed", before=before, after=after)
+ return
+ _collect_public_field_changes(
+ before["__var"],
+ after["__var"],
+ path=path,
+ public_fields=_DIFF_V1_PUBLIC_DEFAULT_ARGS_FIELDS,
+ collector=collector,
+ )
+
+
+def _is_encoded_default_args(value: Any) -> bool:
+ return (
+ isinstance(value, Mapping)
+ and value.get("__type") == "dict"
+ and isinstance(value.get("__var"), Mapping)
+ )
+
+
+def _collect_public_field_changes(
+ before: Mapping[str, Any],
+ after: Mapping[str, Any],
+ *,
+ path: tuple[str, ...],
+ public_fields: frozenset[str] | set[str],
+ collector: _ChangeCollector,
+) -> None:
+ """Walk allowlisted fields one by one and report everything else as a single change."""
+ keys = {str(key) for key in before} | {str(key) for key in after}
+ for key in sorted(keys & public_fields):
+ _collect_changes(
+ before.get(key, _MISSING),
+ after.get(key, _MISSING),
+ path=path + (key,),
+ collector=collector,
+ )
+ if collector.is_truncated:
+ return
+
+ before_custom_fields = {key: before[key] for key in before if key not in public_fields}
+ after_custom_fields = {key: after[key] for key in after if key not in public_fields}
+ if not _is_json_equal(before_custom_fields, after_custom_fields):
+ collector.add(
+ path=path + (_CUSTOM_TASK_FIELDS_PATH_COMPONENT,),
+ operation="changed",
+ before=before_custom_fields,
+ after=after_custom_fields,
+ )
+
+
+def _is_task_mapping_path(path: tuple[str, ...]) -> bool:
+ return path[:2] == ("dag", "tasks") and (
+ len(path) == 3 or (len(path) == 4 and path[3] == "partial_kwargs")
+ )
+
+
+class _TaskGroupPath(NamedTuple):
+ """One reading of the task group path grammar, shared by everything that walks it."""
+
+ field_index: int
+ id_positions: tuple[int, ...]
+ remainder: tuple[str, ...]
+
+
+def _classify_task_group_path(path: tuple[str, ...]) -> _TaskGroupPath | None:
+ """
+ Split a task group path into its nesting prefix and what follows the innermost group.
+
+ The grammar is ``dag/task_group ( /children/<group_id>/1 )* [ /children/<child_id> | /<field> ... ]``:
+ a child entry is the two-element ``["taskgroup", {...}]`` list the serializer writes, so the
+ literal ``1`` selects the nested group mapping inside it. ``field_index`` is where the innermost
+ group's own keys start, ``id_positions`` the group-id components a redacted path must mask, and
+ ``remainder`` the components from ``field_index`` on: ``()`` for the group mapping itself,
+ ``("children",)`` for its children mapping, and a two-element ``("children", child_id)`` for one
+ child entry. Returns ``None`` for a path that is not rooted at the Dag's task group.
+ """
+ if path[:2] != ("dag", "task_group"):
+ return None
+ index = 2
+ id_positions: list[int] = []
+ while len(path) >= index + 2 and path[index] == "children":
+ id_positions.append(index + 1)
+ if len(path) < index + 3 or path[index + 2] != "1":
+ break
+ index += 3
+ return _TaskGroupPath(index, tuple(id_positions), path[index:])
+
+
+def _is_task_group_mapping_path(path: tuple[str, ...]) -> bool:
+ task_group_path = _classify_task_group_path(path)
+ return task_group_path is not None and task_group_path.remainder == ()
+
+
+def _should_recurse_mapping(path: tuple[str, ...]) -> bool:
+ if path in _RECURSIVE_MAPPING_PATHS:
+ return True
+ task_group_path = _classify_task_group_path(path)
+ return task_group_path is not None and task_group_path.remainder == ("children",)
+
+
+def _is_task_group_child_path(path: tuple[str, ...]) -> bool:
+ task_group_path = _classify_task_group_path(path)
+ return (
+ task_group_path is not None
+ and len(task_group_path.remainder) == 2
+ and task_group_path.remainder[0] == "children"
+ )
+
+
+def _get_public_path(path: tuple[str, ...]) -> tuple[str, ...]:
+ if path and path[0] not in _DIFF_V1_PUBLIC_ROOT_FIELDS:
+ # Root is the one level no allowlisted walk covers, so an unrecognised section name would
+ # otherwise be the single path component a redacted caller receives verbatim.
+ return (_CUSTOM_TASK_FIELDS_PATH_COMPONENT,)
+ if (task_group_path := _classify_task_group_path(path)) is not None:
+ public_path = list(path)
+ for position in task_group_path.id_positions:
+ public_path[position] = "*"
+ remainder = task_group_path.remainder
+ if remainder and remainder[0] not in _DIFF_V1_PUBLIC_TASK_GROUP_FIELDS:
+ # Unreachable by construction (_collect_public_field_changes aggregates non-public group
+ # keys first), but truncating like the task branch stops a future caller leaking a key.
+ return (*public_path[: task_group_path.field_index], _CUSTOM_TASK_FIELDS_PATH_COMPONENT)
+ return tuple(public_path)
+ if len(path) >= 3 and path[:2] in _KEYED_COLLECTION_PATHS:
+ path = (*path[:2], "*", *path[3:])
+ if len(path) >= 4 and path[:2] == ("dag", "tasks"):
+ field_index = 4 if path[3] == "partial_kwargs" and len(path) >= 5 else 3
+ public_fields = (
+ _DIFF_V1_PUBLIC_PARTIAL_TASK_FIELDS if field_index == 4 else _DIFF_V1_PUBLIC_TASK_FIELDS
+ )
+ if path[field_index] not in public_fields:
+ return (*path[:field_index], _CUSTOM_TASK_FIELDS_PATH_COMPONENT)
+ return path
+
+
+def _format_path(path: tuple[str, ...]) -> str:
+ return "/" + "/".join(component.replace("~", "~0").replace("/", "~1") for component in path)
+
+
+class _JsonEncodingError(TypeError):
+ """Distinguish unsupported JSON values from comparison implementation errors."""
+
+
+def _serialize_canonical_json(value: Any) -> str:
+ try:
+ return json.dumps(value, ensure_ascii=False, separators=(",", ":"), sort_keys=True)
+ except TypeError as error:
+ raise _JsonEncodingError from error
+
+
+def _is_json_equal(before: Any, after: Any) -> bool:
+ # Compare canonical JSON, not Python ==, so a JSON type change (1 vs 1.0, True vs 1,
+ # False vs 0) registers as a change instead of collapsing under Python equality.
+ return _serialize_canonical_json(before) == _serialize_canonical_json(after)
+
+
+def _get_digest(value: Any) -> str:
+ # SHA-256 provides a content fingerprint, not password protection.
+ # Stored payloads can carry escaped lone surrogates, which strict UTF-8 refuses to encode;
+ # surrogatepass keeps them digestible without altering the bytes of any other payload.
+ canonical_json = _serialize_canonical_json(value).encode("utf-8", errors="surrogatepass")
+ return f"sha256:{hashlib.sha256(canonical_json).hexdigest()}"
Review Comment:
## CodeQL / Use of a broken or weak cryptographic hashing algorithm on sensitive data
[Sensitive data (password)](1) is used in a hashing algorithm (SHA256) that is insecure for password hashing, since it is not a computationally expensive hash function.
[Show more details](https://github.com/apache/airflow/security/code-scanning/656)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add serialized Dag version diff engine [airflow]
Posted by "ephraimbuddy (via GitHub)" <gi...@apache.org>.
ephraimbuddy commented on code in PR #69864:
URL: https://github.com/apache/airflow/pull/69864#discussion_r4018354550
##########
airflow-core/src/airflow/serialization/dag_version_diff.py:
##########
@@ -0,0 +1,898 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""Observed-state diffs for serialized Dag payloads."""
+
+from __future__ import annotations
+
+import copy
+import hashlib
+import json
+from collections.abc import Callable, Mapping
+from datetime import timedelta
+from enum import Enum
+from typing import Any, Literal
+
+import structlog
+
+from airflow.serialization.definitions.baseoperator import SerializedBaseOperator
+from airflow.serialization.definitions.mappedoperator import SerializedMappedOperator
+from airflow.serialization.serialized_objects import (
+ _DAG_CALLBACK_FIELDS,
+ _OPERATOR_TIMEDELTA_FIELDS,
+ DagSerialization,
+ OperatorSerialization,
+)
+
+log = structlog.get_logger(__name__)
+
+DIFF_SCHEMA_VERSION = 1
+DEFAULT_MAX_CHANGES = 500
+MAX_ALLOWED_CHANGES = 5000
+SUPPORTED_SERIALIZED_DAG_SCHEMA_VERSIONS = frozenset((1, 2, 3))
+
+_ORDER_INSENSITIVE_LIST_PATHS = {
+ ("dag", "tags"),
+ ("dag", "allowed_run_types"),
+}
+_KEYED_COLLECTION_PATHS = {
+ ("dag", "tasks"),
+ ("dag", "dag_dependencies"),
+ *_ORDER_INSENSITIVE_LIST_PATHS,
+}
+_CUSTOM_TASK_FIELDS_PATH_COMPONENT = "custom_fields"
+# This allowlist is part of diff schema v1. Serializer schema changes must not
+# silently change the paths visible to callers of the diff API.
+_DIFF_V1_PUBLIC_TASK_FIELDS = frozenset(
Review Comment:
Fixed. All eight fields are now public: `python_callable_name`, `expand_input`, `op_kwargs_expand_input`, `resources`, `run_as_user`, `email`, `_operator_name` and `has_retry_policy`. A renamed `@task` callable now reports `/dag/tasks/*/python_callable_name` instead of an unnamed `custom_fields` record.
The test is pinned to a real `to_dict()` payload as you suggested, asserting the allowlist both ways so it cannot drift again — it caught `is_stub` sitting in the same position shortly after. It is still only as complete as the operator shapes the sweep builds, so a new shape needs a new task there, not just a new entry.
---
Drafted-by: Claude Code (Opus 5); reviewed by @ephraimbuddy before posting
##########
airflow-core/src/airflow/serialization/dag_version_diff.py:
##########
@@ -0,0 +1,898 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""Observed-state diffs for serialized Dag payloads."""
+
+from __future__ import annotations
+
+import copy
+import hashlib
+import json
+from collections.abc import Callable, Mapping
+from datetime import timedelta
+from enum import Enum
+from typing import Any, Literal
+
+import structlog
+
+from airflow.serialization.definitions.baseoperator import SerializedBaseOperator
+from airflow.serialization.definitions.mappedoperator import SerializedMappedOperator
+from airflow.serialization.serialized_objects import (
+ _DAG_CALLBACK_FIELDS,
+ _OPERATOR_TIMEDELTA_FIELDS,
+ DagSerialization,
+ OperatorSerialization,
+)
+
+log = structlog.get_logger(__name__)
+
+DIFF_SCHEMA_VERSION = 1
+DEFAULT_MAX_CHANGES = 500
+MAX_ALLOWED_CHANGES = 5000
+SUPPORTED_SERIALIZED_DAG_SCHEMA_VERSIONS = frozenset((1, 2, 3))
+
+_ORDER_INSENSITIVE_LIST_PATHS = {
+ ("dag", "tags"),
+ ("dag", "allowed_run_types"),
+}
+_KEYED_COLLECTION_PATHS = {
+ ("dag", "tasks"),
+ ("dag", "dag_dependencies"),
+ *_ORDER_INSENSITIVE_LIST_PATHS,
+}
+_CUSTOM_TASK_FIELDS_PATH_COMPONENT = "custom_fields"
+# This allowlist is part of diff schema v1. Serializer schema changes must not
+# silently change the paths visible to callers of the diff API.
+_DIFF_V1_PUBLIC_TASK_FIELDS = frozenset(
+ {
+ "__type",
+ "_disallow_kwargs_override",
+ "_expand_input_attr",
+ "_is_mapped",
+ "_is_sensor",
+ "_logger_name",
+ "_needs_expansion",
+ "_operator_extra_links",
+ "_task_display_name",
+ "_task_module",
+ "allow_nested_operators",
+ "depends_on_past",
+ "do_xcom_push",
+ "doc",
+ "doc_json",
+ "doc_md",
+ "doc_rst",
+ "doc_yaml",
+ "downstream_task_ids",
+ "email_on_failure",
+ "email_on_retry",
+ "end_date",
+ "execution_timeout",
+ "executor",
+ "executor_config",
+ "has_on_execute_callback",
+ "has_on_failure_callback",
+ "has_on_retry_callback",
+ "has_on_skipped_callback",
+ "has_on_success_callback",
+ "ignore_first_depends_on_past",
+ "inlets",
+ "is_setup",
+ "is_teardown",
+ "map_index_template",
+ "max_active_tis_per_dag",
+ "max_active_tis_per_dagrun",
+ "max_retry_delay",
+ "multiple_outputs",
+ "on_failure_fail_dagrun",
+ "outlets",
+ "owner",
+ "params",
+ "partial_kwargs",
+ "pool",
+ "pool_slots",
+ "priority_weight",
+ "queue",
+ "render_template_as_native_obj",
+ "retries",
+ "retry_delay",
+ "retry_exponential_backoff",
+ "start_date",
+ "start_from_trigger",
+ "start_trigger_args",
+ "task_id",
+ "task_type",
+ "template_ext",
+ "template_fields",
+ "template_fields_renderers",
+ "trigger_rule",
+ "ui_color",
+ "ui_fgcolor",
+ "wait_for_downstream",
+ "wait_for_past_depends_before_skipping",
+ "weight_rule",
+ }
+)
+_DIFF_V1_REDACTED_SCHEMA_TASK_FIELDS = frozenset({"_arg_bindings"})
+# _get_category classifies task fields with these; every name must be a public task field or the
+# entry is unreachable, since a non-public field is aggregated under custom_fields before lookup.
+_DIFF_V1_TASK_ASSET_FIELDS = frozenset({"inlets", "outlets"})
+_DIFF_V1_TASK_PARAM_FIELDS = frozenset({"params"})
+_DIFF_V1_TASK_DEPENDENCY_FIELDS = frozenset({"downstream_task_ids"})
+_DIFF_V1_TASK_METADATA_FIELDS = frozenset(
+ {
+ "doc",
+ "doc_json",
+ "doc_md",
+ "doc_rst",
+ "doc_yaml",
+ "owner",
+ "ui_color",
+ "ui_fgcolor",
+ "_task_display_name",
+ "task_display_name",
+ }
+)
+_DIFF_V1_PUBLIC_PARTIAL_TASK_FIELDS = _DIFF_V1_PUBLIC_TASK_FIELDS | {"task_display_name"}
+# Classify every Dag schema field explicitly so new fields require a policy decision.
+_DIFF_V1_DAG_FIELD_CATEGORIES = {
+ "_concurrency": "schedule",
+ "_processor_dags_folder": "provenance",
+ "access_control": "authorization",
+ "allowed_run_types": "schedule",
+ "bundle_name": "provenance",
+ "catchup": "schedule",
+ "dag_dependencies": "dependency",
+ "dag_display_name": "metadata",
+ "dag_id": "metadata",
+ "dagrun_timeout": "schedule",
+ "deadline": "deadline",
+ "default_args": "param",
+ "description": "metadata",
+ "disable_bundle_versioning": "task",
+ "doc_md": "metadata",
+ "edge_info": "metadata",
+ "end_date": "schedule",
+ "fail_fast": "schedule",
+ "fileloc": "provenance",
+ "has_on_failure_callback": "callback",
+ "has_on_success_callback": "callback",
+ "is_paused_upon_creation": "schedule",
+ "max_active_runs": "schedule",
+ "max_active_tasks": "schedule",
+ "max_consecutive_failed_dag_runs": "schedule",
+ "owner_links": "metadata",
+ "params": "param",
+ "relative_fileloc": "provenance",
+ "render_template_as_native_obj": "task",
+ "rerun_with_latest_version": "task",
+ "start_date": "schedule",
+ "tags": "metadata",
+ "task_group": "task",
+ "tasks": "task",
+ "timetable": "schedule",
+ "timezone": "schedule",
+}
+_DIFF_V1_LEGACY_DAG_FIELD_CATEGORIES = {
+ "fail_stop": "schedule",
+ "on_failure_callback": "callback",
+ "on_success_callback": "callback",
+ "schedule": "schedule",
+ "schedule_interval": "schedule",
+}
+_RECURSIVE_MAPPING_PATHS = {
+ (),
+ ("dag",),
+ ("provenance",),
+ *_KEYED_COLLECTION_PATHS,
+}
+_DIFF_V1_TASK_GROUP_METADATA_FIELDS = frozenset(
+ {"group_display_name", "tooltip", "doc_md", "ui_color", "ui_fgcolor"}
+)
+_DIFF_V1_PUBLIC_TASK_GROUP_FIELDS = _DIFF_V1_TASK_GROUP_METADATA_FIELDS | {
+ "_group_id",
+ "prefix_group_id",
+ "children",
+ "upstream_group_ids",
+ "downstream_group_ids",
+ "upstream_task_ids",
+ "downstream_task_ids",
+ "expand_input",
+ "is_mapped",
+}
+
+
+def build_unavailable_dag_diff(
+ *,
+ base_data: dict[str, Any] | None,
+ target_data: dict[str, Any] | None,
+ reason: str,
+) -> dict[str, Any]:
+ """Report a known unavailable reason without comparing the stored payloads."""
+ return _mark_unavailable(
+ _build_diff_result(_get_schema_version(base_data), _get_schema_version(target_data)), reason
+ )
+
+
+def build_serialized_dag_diff(
+ *,
+ base_data: dict[str, Any] | None,
+ target_data: dict[str, Any] | None,
+ base_provenance: Mapping[str, Any] | None = None,
+ target_provenance: Mapping[str, Any] | None = None,
+ include_values: bool = False,
+ max_changes: int = DEFAULT_MAX_CHANGES,
+) -> dict[str, Any]:
+ """
+ Build a bounded, deterministic diff from two stored serialized Dag payloads.
+
+ Raw values, digests, and value-derived path components are returned only when
+ ``include_values`` is true. Callers must authorize disclosure of the entire
+ serialized payload, including access-control role names and permission mappings,
+ before enabling it.
+ """
+ validate_max_changes(max_changes)
+
+ base_schema_version = _get_schema_version(base_data)
+ target_schema_version = _get_schema_version(target_data)
+ result = _build_diff_result(base_schema_version, target_schema_version)
+
+ if base_data is None or target_data is None:
+ return _mark_unavailable(result, "serialized_dag_missing")
+
+ if base_schema_version is None or target_schema_version is None:
+ return _mark_unavailable(result, "serialized_dag_schema_version_missing")
+
+ unsupported_versions = [
+ version
+ for version in (base_schema_version, target_schema_version)
+ if version not in SUPPORTED_SERIALIZED_DAG_SCHEMA_VERSIONS
+ ]
+ if unsupported_versions:
+ return _mark_unavailable(
+ result, f"unsupported_serialized_dag_schema_version:{unsupported_versions[0]}"
+ )
+
+ try:
+ base_document = _canonicalize_payload_v1(base_data)
+ target_document = _canonicalize_payload_v1(target_data)
+ base_document["provenance"] = _canonicalize_value(dict(base_provenance or {}), path=("provenance",))
+ target_document["provenance"] = _canonicalize_value(
+ dict(target_provenance or {}), path=("provenance",)
+ )
+ except (AttributeError, KeyError, OverflowError, TypeError, ValueError) as error:
Review Comment:
Fixed at both sites: `RecursionError` is in the canonicalization tuple, and `_collect_changes` has its own handler returning `serialized_dag_recursion_limit_exceeded`.
Your trace also exposed an adjacent hole. Reading `SerializedDagModel.data` decompresses and parses the row, and that read sat outside any boundary, so unparseable compressed JSON, truncated bytes (`zlib.error`) and deep nesting all escaped while the uncompressed equivalent degraded. It is now inside the boundary with a distinct `serialized_dag_decode_failed`.
---
Drafted-by: Claude Code (Opus 5); reviewed by @ephraimbuddy before posting
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add serialized Dag version diff engine [airflow]
Posted by "ephraimbuddy (via GitHub)" <gi...@apache.org>.
ephraimbuddy commented on code in PR #69864:
URL: https://github.com/apache/airflow/pull/69864#discussion_r4018358185
##########
airflow-core/src/airflow/serialization/dag_version_diff.py:
##########
@@ -0,0 +1,898 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""Observed-state diffs for serialized Dag payloads."""
+
+from __future__ import annotations
+
+import copy
+import hashlib
+import json
+from collections.abc import Callable, Mapping
+from datetime import timedelta
+from enum import Enum
+from typing import Any, Literal
+
+import structlog
+
+from airflow.serialization.definitions.baseoperator import SerializedBaseOperator
+from airflow.serialization.definitions.mappedoperator import SerializedMappedOperator
+from airflow.serialization.serialized_objects import (
+ _DAG_CALLBACK_FIELDS,
+ _OPERATOR_TIMEDELTA_FIELDS,
+ DagSerialization,
+ OperatorSerialization,
+)
+
+log = structlog.get_logger(__name__)
+
+DIFF_SCHEMA_VERSION = 1
+DEFAULT_MAX_CHANGES = 500
+MAX_ALLOWED_CHANGES = 5000
+SUPPORTED_SERIALIZED_DAG_SCHEMA_VERSIONS = frozenset((1, 2, 3))
+
+_ORDER_INSENSITIVE_LIST_PATHS = {
+ ("dag", "tags"),
+ ("dag", "allowed_run_types"),
+}
+_KEYED_COLLECTION_PATHS = {
+ ("dag", "tasks"),
+ ("dag", "dag_dependencies"),
+ *_ORDER_INSENSITIVE_LIST_PATHS,
+}
+_CUSTOM_TASK_FIELDS_PATH_COMPONENT = "custom_fields"
+# This allowlist is part of diff schema v1. Serializer schema changes must not
+# silently change the paths visible to callers of the diff API.
+_DIFF_V1_PUBLIC_TASK_FIELDS = frozenset(
+ {
+ "__type",
+ "_disallow_kwargs_override",
+ "_expand_input_attr",
+ "_is_mapped",
+ "_is_sensor",
+ "_logger_name",
+ "_needs_expansion",
+ "_operator_extra_links",
+ "_task_display_name",
+ "_task_module",
+ "allow_nested_operators",
+ "depends_on_past",
+ "do_xcom_push",
+ "doc",
+ "doc_json",
+ "doc_md",
+ "doc_rst",
+ "doc_yaml",
+ "downstream_task_ids",
+ "email_on_failure",
+ "email_on_retry",
+ "end_date",
+ "execution_timeout",
+ "executor",
+ "executor_config",
+ "has_on_execute_callback",
+ "has_on_failure_callback",
+ "has_on_retry_callback",
+ "has_on_skipped_callback",
+ "has_on_success_callback",
+ "ignore_first_depends_on_past",
+ "inlets",
+ "is_setup",
+ "is_teardown",
+ "map_index_template",
+ "max_active_tis_per_dag",
+ "max_active_tis_per_dagrun",
+ "max_retry_delay",
+ "multiple_outputs",
+ "on_failure_fail_dagrun",
+ "outlets",
+ "owner",
+ "params",
+ "partial_kwargs",
+ "pool",
+ "pool_slots",
+ "priority_weight",
+ "queue",
+ "render_template_as_native_obj",
+ "retries",
+ "retry_delay",
+ "retry_exponential_backoff",
+ "start_date",
+ "start_from_trigger",
+ "start_trigger_args",
+ "task_id",
+ "task_type",
+ "template_ext",
+ "template_fields",
+ "template_fields_renderers",
+ "trigger_rule",
+ "ui_color",
+ "ui_fgcolor",
+ "wait_for_downstream",
+ "wait_for_past_depends_before_skipping",
+ "weight_rule",
+ }
+)
+_DIFF_V1_REDACTED_SCHEMA_TASK_FIELDS = frozenset({"_arg_bindings"})
+# _get_category classifies task fields with these; every name must be a public task field or the
+# entry is unreachable, since a non-public field is aggregated under custom_fields before lookup.
+_DIFF_V1_TASK_ASSET_FIELDS = frozenset({"inlets", "outlets"})
+_DIFF_V1_TASK_PARAM_FIELDS = frozenset({"params"})
+_DIFF_V1_TASK_DEPENDENCY_FIELDS = frozenset({"downstream_task_ids"})
+_DIFF_V1_TASK_METADATA_FIELDS = frozenset(
+ {
+ "doc",
+ "doc_json",
+ "doc_md",
+ "doc_rst",
+ "doc_yaml",
+ "owner",
+ "ui_color",
+ "ui_fgcolor",
+ "_task_display_name",
+ "task_display_name",
+ }
+)
+_DIFF_V1_PUBLIC_PARTIAL_TASK_FIELDS = _DIFF_V1_PUBLIC_TASK_FIELDS | {"task_display_name"}
+# Classify every Dag schema field explicitly so new fields require a policy decision.
+_DIFF_V1_DAG_FIELD_CATEGORIES = {
+ "_concurrency": "schedule",
+ "_processor_dags_folder": "provenance",
+ "access_control": "authorization",
+ "allowed_run_types": "schedule",
+ "bundle_name": "provenance",
+ "catchup": "schedule",
+ "dag_dependencies": "dependency",
+ "dag_display_name": "metadata",
+ "dag_id": "metadata",
+ "dagrun_timeout": "schedule",
+ "deadline": "deadline",
+ "default_args": "param",
+ "description": "metadata",
+ "disable_bundle_versioning": "task",
+ "doc_md": "metadata",
+ "edge_info": "metadata",
+ "end_date": "schedule",
+ "fail_fast": "schedule",
+ "fileloc": "provenance",
+ "has_on_failure_callback": "callback",
+ "has_on_success_callback": "callback",
+ "is_paused_upon_creation": "schedule",
+ "max_active_runs": "schedule",
+ "max_active_tasks": "schedule",
+ "max_consecutive_failed_dag_runs": "schedule",
+ "owner_links": "metadata",
+ "params": "param",
+ "relative_fileloc": "provenance",
+ "render_template_as_native_obj": "task",
+ "rerun_with_latest_version": "task",
+ "start_date": "schedule",
+ "tags": "metadata",
+ "task_group": "task",
+ "tasks": "task",
+ "timetable": "schedule",
+ "timezone": "schedule",
+}
+_DIFF_V1_LEGACY_DAG_FIELD_CATEGORIES = {
+ "fail_stop": "schedule",
+ "on_failure_callback": "callback",
+ "on_success_callback": "callback",
+ "schedule": "schedule",
+ "schedule_interval": "schedule",
+}
+_RECURSIVE_MAPPING_PATHS = {
+ (),
+ ("dag",),
+ ("provenance",),
+ *_KEYED_COLLECTION_PATHS,
+}
+_DIFF_V1_TASK_GROUP_METADATA_FIELDS = frozenset(
+ {"group_display_name", "tooltip", "doc_md", "ui_color", "ui_fgcolor"}
+)
+_DIFF_V1_PUBLIC_TASK_GROUP_FIELDS = _DIFF_V1_TASK_GROUP_METADATA_FIELDS | {
+ "_group_id",
+ "prefix_group_id",
+ "children",
+ "upstream_group_ids",
+ "downstream_group_ids",
+ "upstream_task_ids",
+ "downstream_task_ids",
+ "expand_input",
+ "is_mapped",
+}
+
+
+def build_unavailable_dag_diff(
+ *,
+ base_data: dict[str, Any] | None,
+ target_data: dict[str, Any] | None,
+ reason: str,
+) -> dict[str, Any]:
+ """Report a known unavailable reason without comparing the stored payloads."""
+ return _mark_unavailable(
+ _build_diff_result(_get_schema_version(base_data), _get_schema_version(target_data)), reason
+ )
+
+
+def build_serialized_dag_diff(
+ *,
+ base_data: dict[str, Any] | None,
+ target_data: dict[str, Any] | None,
+ base_provenance: Mapping[str, Any] | None = None,
+ target_provenance: Mapping[str, Any] | None = None,
+ include_values: bool = False,
+ max_changes: int = DEFAULT_MAX_CHANGES,
+) -> dict[str, Any]:
+ """
+ Build a bounded, deterministic diff from two stored serialized Dag payloads.
+
+ Raw values, digests, and value-derived path components are returned only when
+ ``include_values`` is true. Callers must authorize disclosure of the entire
+ serialized payload, including access-control role names and permission mappings,
+ before enabling it.
+ """
+ validate_max_changes(max_changes)
+
+ base_schema_version = _get_schema_version(base_data)
+ target_schema_version = _get_schema_version(target_data)
+ result = _build_diff_result(base_schema_version, target_schema_version)
+
+ if base_data is None or target_data is None:
+ return _mark_unavailable(result, "serialized_dag_missing")
+
+ if base_schema_version is None or target_schema_version is None:
+ return _mark_unavailable(result, "serialized_dag_schema_version_missing")
+
+ unsupported_versions = [
+ version
+ for version in (base_schema_version, target_schema_version)
+ if version not in SUPPORTED_SERIALIZED_DAG_SCHEMA_VERSIONS
+ ]
+ if unsupported_versions:
+ return _mark_unavailable(
+ result, f"unsupported_serialized_dag_schema_version:{unsupported_versions[0]}"
+ )
+
+ try:
+ base_document = _canonicalize_payload_v1(base_data)
+ target_document = _canonicalize_payload_v1(target_data)
+ base_document["provenance"] = _canonicalize_value(dict(base_provenance or {}), path=("provenance",))
+ target_document["provenance"] = _canonicalize_value(
+ dict(target_provenance or {}), path=("provenance",)
+ )
+ except (AttributeError, KeyError, OverflowError, TypeError, ValueError) as error:
+ log.warning(
+ "Serialized Dag diff canonicalization failed",
+ error_type=type(error).__name__,
+ base_schema_version=base_schema_version,
+ target_schema_version=target_schema_version,
+ )
+ return _mark_unavailable(result, "serialized_dag_canonicalization_failed")
+
+ collector = _ChangeCollector(max_changes=max_changes, include_values=include_values)
+ try:
+ _collect_changes(base_document, target_document, path=(), collector=collector)
+ except _JsonEncodingError:
+ log.warning(
+ "Serialized Dag diff JSON encoding failed",
+ base_schema_version=base_schema_version,
+ target_schema_version=target_schema_version,
+ )
+ return _mark_unavailable(result, "serialized_dag_json_encoding_failed")
+
+ result["changes"] = collector.changes
+ result["truncated"] = collector.is_truncated
+ if include_values:
+ result["values"] = {"status": "available"}
+ return result
+
+
+class _ChangeCollector:
+ def __init__(self, *, max_changes: int, include_values: bool) -> None:
+ self.changes: list[dict[str, Any]] = []
+ self.count = 0
+ self.max_changes = max_changes
+ self.include_values = include_values
+
+ @property
+ def is_truncated(self) -> bool:
+ return self.count > self.max_changes
+
+ def add(
+ self,
+ *,
+ path: tuple[str, ...],
+ operation: Literal["added", "removed", "changed"],
+ before: Any,
+ after: Any,
+ ) -> None:
+ self.count += 1
+ if len(self.changes) >= self.max_changes:
+ return
+
+ public_path = _get_public_path(path)
+ category = _get_category(public_path)
+ change: dict[str, Any] = {
+ "path": _format_path(path if self.include_values else public_path),
Review Comment:
Done. Records sharing a public path and operation now merge into one carrying an `occurrence_count`. Your 600-task case at `max_changes=500`:
```
redacted: 3 records, truncated=False, {start_date: 1,
tasks/*/retries: 600, tasks/*/start_date: 600}
authorized: 500 records, truncated=True
```
Grouping alone left the counts wrong (250/249, `truncated=True` with all three paths present) because the bound was applied before the grouping lookup. Counting now continues for a path already in the result, so counts are exact and `truncated` means only that an unseen path was dropped. The docstring contract and the misleading test name were corrected with it.
---
Drafted-by: Claude Code (Opus 5); reviewed by @ephraimbuddy before posting
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add serialized Dag version diff engine [airflow]
Posted by "ephraimbuddy (via GitHub)" <gi...@apache.org>.
ephraimbuddy commented on code in PR #69864:
URL: https://github.com/apache/airflow/pull/69864#discussion_r4030027592
##########
airflow-core/src/airflow/serialization/dag_version_diff.py:
##########
@@ -0,0 +1,1222 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""
+Observed-state diffs for serialized Dag payloads.
+
+Diff schema v1 -- the wire format every entry point here returns.
+:func:`build_serialized_dag_diff` and :func:`build_unavailable_dag_diff` both return one
+dictionary in the shape described below, and ``DIFF_SCHEMA_VERSION`` is the version of that
+shape: bump it whenever a client-observable part of this contract changes.
+
+Top-level keys, all always present:
+
+* ``diff_schema_version`` -- ``int``, the value of ``DIFF_SCHEMA_VERSION``.
+* ``serialized_dag_schema_versions`` -- ``{"base": int | None, "target": int | None}``, the
+ ``__version`` each stored payload carried. ``None`` means the key was absent or not an
+ integer; ``bool`` is rejected, so ``__version: true`` reads as ``None`` and never as
+ version 1. A version in ``SUPPORTED_SERIALIZED_DAG_SCHEMA_VERSIONS`` is upgraded to the
+ newest supported version before anything is compared.
+* ``mode`` -- ``"observed_state"`` when the two payloads were compared, ``"unavailable"``
+ when no comparison happened.
+* ``changes`` -- the change records described below, ordered by the deterministic walk.
+ Always empty when ``mode`` is ``"unavailable"``.
+* ``truncated`` -- ``True`` when more underlying changes exist than ``max_changes`` allowed.
+ Always ``False`` when ``mode`` is ``"unavailable"``.
+* ``values`` -- ``{"status": "available" | "unavailable"}``. ``"available"`` only for an
+ ``"observed_state"`` result whose caller authorized value disclosure, so
+ ``result["values"]["status"]`` is safe to read on every result of every entry point.
+
+``unavailable_reason`` is the one conditional key: present exactly when ``mode`` is
+``"unavailable"``. :func:`build_serialized_dag_diff` produces:
+
+* ``serialized_dag_missing`` -- one side has no stored payload.
+* ``serialized_dag_schema_version_missing`` -- one side carries no usable ``__version``.
+* ``unsupported_serialized_dag_schema_version:<n>`` -- ``<n>`` is the first unsupported
+ version found, base before target.
+* ``serialized_dag_canonicalization_failed`` -- a payload could not be normalized (malformed
+ structure, an unsupported ``client_defaults`` section, an unencodable value).
+* ``serialized_dag_recursion_limit_exceeded`` -- the comparison walk ran too deep.
+* ``serialized_dag_json_encoding_failed`` -- a value could not be encoded as canonical JSON.
+
+:meth:`~airflow.models.dag_version.DagVersion.get_diff` reuses
+``serialized_dag_decode_failed``, for a stored payload that will not decompress or parse, and
+``deadline_alert_missing``, for a payload that references a deadline alert row that is gone.
+:func:`build_unavailable_dag_diff` passes its caller's reason through unchanged, so a new
+caller extends this list rather than inventing an undocumented value.
+
+Each change record carries these keys in both disclosure modes:
+
+* ``path`` -- a JSON-Pointer-style path into a synthetic document with two roots: ``/dag``
+ (the canonicalized serialized Dag, with ``__version`` dropped and client defaults folded
+ into the tasks) and ``/provenance`` (the provenance mapping the caller supplied --
+ ``bundle_name``, ``bundle_version`` and ``version_data`` from ``get_diff``). ``~`` and
+ ``/`` inside a component are escaped as ``~0`` and ``~1``.
+* ``operation`` -- ``"added"``, ``"removed"`` or ``"changed"``.
+* ``category`` -- a :data:`DiffCategory`: ``task`` for task definitions and the Dag fields
+ that shape task execution, ``dependency`` for ``dag_dependencies`` and downstream task
+ ids, ``schedule`` for timetable, dates and concurrency, ``param`` for params,
+ ``asset`` for task inlets and outlets, ``deadline`` for deadline alerts,
+ ``callback`` for callback presence, ``metadata`` for descriptive and display fields,
+ ``authorization`` for ``access_control``, ``provenance`` for everything under
+ ``/provenance`` plus the file and bundle locators, and ``unknown`` for anything
+ unclassified. ``_DIFF_V1_DAG_FIELD_CATEGORIES`` is the authoritative Dag-field mapping;
+ an unclassified task field falls back to ``task``, which over-reports impact.
+ ``default_args`` is the one Dag field classified per key rather than as a whole, because its
+ keys are the operator fields they will be applied to: an allowlisted key takes that field's
+ category (``owner`` is ``metadata``, ``retries`` is ``task``), and the aggregate record for
+ the rest keeps the conservative ``task``. Only a stored value that is not a readable dict
+ envelope classifies as ``task`` as a whole.
+* ``impact`` -- a :data:`DiffImpact` derived from ``category``: ``provenance``, ``metadata``
+ and ``authorization`` carry through unchanged, every operational category becomes
+ ``execution``, and ``unknown`` stays ``unknown``.
+* ``occurrence_count`` -- how many underlying changes the record stands for.
+
+A caller that authorized values (``include_values=True``, ``values.status`` ``"available"``)
+additionally gets:
+
+* ``before_digest`` / ``after_digest`` -- ``"sha256:<hex>"`` over the canonical JSON of that
+ side, or ``None`` when that side is missing. Both keys are always present.
+* ``before_value`` / ``after_value`` -- the canonicalized value, which is the stored one after
+ schema and client defaults are folded in and params are normalized. ``before_value`` is absent from
+ an ``added`` record and ``after_value`` from a ``removed`` one, which is how a missing side
+ is told apart from a stored ``null``.
+
+``path`` and ``occurrence_count`` also differ between the modes. A redacted record reports the
+public path: members of keyed collections (``/dag/tasks``, ``/dag/dag_dependencies``,
+``/dag/tags``, ``/dag/allowed_run_types`` and task group children) collapse to ``*`` instead
+of naming a task, tag or group, and records sharing a public path and operation are merged
+into one whose ``occurrence_count`` counts the merged changes. An authorized record keeps the
+identifying component and always has an ``occurrence_count`` of 1. Fields outside the v1 task,
+task group and ``default_args`` allowlists are aggregated into a single ``custom_fields`` record
+in both modes, so neither a private serializer field nor a user-chosen ``default_args`` key is
+ever named by either -- ``/dag/default_args`` names only allowlisted operator field names, which
+is why its keys can be classified at all. A top-level key outside the v1 root allowlist is
+likewise reported as ``/custom_fields`` when redacted, while an authorized record keeps its real
+root path.
+
+``max_changes`` bounds how many underlying changes either mode admits, so authorization decides
+what a change carries and never which changes exist. Admission is what the bound limits, not
+the record count: a redacted diff can stop admitting while holding fewer records than the
+bound, because several admitted changes merged into one record. A change reaching a group
+already in the result is counted without consuming a new record, so repeats keep accumulating
+until a change that needs a new record stops the walk.
+
+``truncated`` reports that such a change was dropped. When it is ``False`` every changed path
+is present with an exact ``occurrence_count``. When it is ``True`` the walk stopped early: some
+paths are absent rather than unchanged, and every ``occurrence_count`` is a lower bound, since
+occurrences after the stop were never reached.
+
+Typed dictionaries for the result and the change record are deliberately deferred -- the
+record shape varies with disclosure mode, so modelling it belongs with the REST layer that
+exposes it. :data:`DiffCategory` and :data:`DiffImpact` are stable enough to generate enums
+from today.
+"""
+
+from __future__ import annotations
+
+import copy
+import hashlib
+import json
+from collections.abc import Callable, Mapping, MutableMapping
+from contextlib import suppress
+from dataclasses import dataclass
+from datetime import timedelta
+from enum import Enum
+from typing import Any, Literal, NamedTuple
+
+import structlog
+
+from airflow.serialization.definitions.baseoperator import SerializedBaseOperator
+from airflow.serialization.definitions.mappedoperator import SerializedMappedOperator
+from airflow.serialization.serialized_objects import (
+ _DAG_CALLBACK_FIELDS,
+ _OPERATOR_TIMEDELTA_FIELDS,
+ DagSerialization,
+ OperatorSerialization,
+)
+
+log = structlog.get_logger(__name__)
+
+DIFF_SCHEMA_VERSION = 1
+DEFAULT_MAX_CHANGES = 500
+MAX_ALLOWED_CHANGES = 5000
+SUPPORTED_SERIALIZED_DAG_SCHEMA_VERSIONS = frozenset((1, 2, 3))
+
+DiffCategory = Literal[
+ "asset",
+ "authorization",
+ "callback",
+ "deadline",
+ "dependency",
+ "metadata",
+ "param",
+ "provenance",
+ "schedule",
+ "task",
+ "unknown",
+]
+DiffImpact = Literal["authorization", "execution", "metadata", "provenance", "unknown"]
+
+_ORDER_INSENSITIVE_LIST_PATHS = {
+ ("dag", "tags"),
+ ("dag", "allowed_run_types"),
+}
+_KEYED_COLLECTION_PATHS = {
+ ("dag", "tasks"),
+ ("dag", "dag_dependencies"),
+ *_ORDER_INSENSITIVE_LIST_PATHS,
+}
+# Canonicalization folds away __version and client_defaults and adds provenance, so these are the
+# only top-level sections the walk is allowed to name.
+_DIFF_V1_PUBLIC_ROOT_FIELDS = frozenset({"dag", "provenance"})
+_CUSTOM_TASK_FIELDS_PATH_COMPONENT = "custom_fields"
+# This allowlist is part of diff schema v1. Serializer schema changes must not
+# silently change the paths visible to callers of the diff API.
+_DIFF_V1_PUBLIC_TASK_FIELDS = frozenset(
+ {
+ "__type",
+ "_can_skip_downstream",
+ "_disallow_kwargs_override",
+ "_expand_input_attr",
+ "_is_empty",
+ "_is_mapped",
+ "_is_sensor",
+ "_logger_name",
+ "_needs_expansion",
+ "_operator_extra_links",
+ "_operator_name",
+ "_task_display_name",
+ "_task_module",
+ "allow_nested_operators",
+ "depends_on_past",
+ "do_xcom_push",
+ "doc",
+ "doc_json",
+ "doc_md",
+ "doc_rst",
+ "doc_yaml",
+ "downstream_task_ids",
+ "email",
+ "email_on_failure",
+ "email_on_retry",
+ "end_date",
+ "execution_timeout",
+ "executor",
+ "executor_config",
+ "expand_input",
+ "has_on_execute_callback",
+ "has_on_failure_callback",
+ "has_on_retry_callback",
+ "has_on_skipped_callback",
+ "has_on_success_callback",
+ "has_retry_policy",
+ "ignore_first_depends_on_past",
+ "inlets",
+ "is_setup",
+ "is_stub",
+ "is_teardown",
+ "map_index_template",
+ "max_active_tis_per_dag",
+ "max_active_tis_per_dagrun",
+ "max_retry_delay",
+ "multiple_outputs",
+ "on_failure_fail_dagrun",
+ "op_kwargs_expand_input",
+ "outlets",
+ "owner",
+ "params",
+ "partial_kwargs",
+ "pool",
+ "pool_slots",
+ "priority_weight",
+ "python_callable_name",
+ "queue",
+ "render_template_as_native_obj",
+ "reschedule",
+ "resources",
+ "retries",
+ "retry_delay",
+ "retry_exponential_backoff",
+ "run_as_user",
+ "start_date",
+ "start_from_trigger",
+ "start_trigger_args",
+ "task_id",
+ "task_type",
+ "template_ext",
+ "template_fields",
+ "template_fields_renderers",
+ "trigger_rule",
+ "ui_color",
+ "ui_fgcolor",
+ "wait_for_downstream",
+ "wait_for_past_depends_before_skipping",
+ "weight_rule",
+ }
+)
+_DIFF_V1_REDACTED_SCHEMA_TASK_FIELDS = frozenset({"_arg_bindings"})
+# Task-field categories used by _get_category.
+_DIFF_V1_TASK_ASSET_FIELDS = frozenset({"inlets", "outlets"})
+_DIFF_V1_TASK_PARAM_FIELDS = frozenset({"params"})
+_DIFF_V1_TASK_DEPENDENCY_FIELDS = frozenset({"downstream_task_ids"})
+_DIFF_V1_TASK_METADATA_FIELDS = frozenset(
+ {
+ "_operator_name",
+ "doc",
+ "doc_json",
+ "doc_md",
+ "doc_rst",
+ "doc_yaml",
+ "owner",
+ "ui_color",
+ "ui_fgcolor",
+ "_task_display_name",
+ "task_display_name",
+ }
+)
+_DIFF_V1_PUBLIC_PARTIAL_TASK_FIELDS = _DIFF_V1_PUBLIC_TASK_FIELDS | {"task_display_name"}
+_DEFAULT_ARGS_PATH = ("dag", "default_args")
+# A Dag's default_args holds the same operator __init__ kwargs partial_kwargs holds -- the
+# serializer rewrites _HAS_FLAG_FIELDS in both -- so it reuses that allowlist. "__type" names the
+# stored value's own dict envelope rather than a key inside it.
+_DIFF_V1_PUBLIC_DEFAULT_ARGS_FIELDS = _DIFF_V1_PUBLIC_PARTIAL_TASK_FIELDS - {"__type"}
+# Classify every Dag schema field explicitly so new fields require a policy decision.
+_DIFF_V1_DAG_FIELD_CATEGORIES: dict[str, DiffCategory] = {
+ "_concurrency": "schedule",
+ "_processor_dags_folder": "provenance",
+ "access_control": "authorization",
+ "allowed_run_types": "schedule",
+ "bundle_name": "provenance",
+ "catchup": "schedule",
+ "dag_dependencies": "dependency",
+ "dag_display_name": "metadata",
+ "dag_id": "metadata",
+ "dagrun_timeout": "schedule",
+ "deadline": "deadline",
+ # Only reached when the stored value is not the dict envelope _collect_default_args_changes
+ # walks per key; an unreadable envelope keeps the conservative impact of its widest key.
+ "default_args": "task",
+ "description": "metadata",
+ # Decides whether a run pins a bundle version, so it changes which code later runs execute
+ # and whether triggering a historical version is allowed at all.
+ "disable_bundle_versioning": "task",
+ "doc_md": "metadata",
+ "edge_info": "metadata",
+ "end_date": "schedule",
+ "fail_fast": "schedule",
+ "fileloc": "provenance",
+ "has_on_failure_callback": "callback",
+ "has_on_success_callback": "callback",
+ "is_paused_upon_creation": "schedule",
+ "max_active_runs": "schedule",
+ "max_active_tasks": "schedule",
+ "max_consecutive_failed_dag_runs": "schedule",
+ "owner_links": "metadata",
+ "params": "param",
+ "relative_fileloc": "provenance",
+ "render_template_as_native_obj": "task",
+ "rerun_with_latest_version": "task",
+ "start_date": "schedule",
+ "tags": "metadata",
+ "task_group": "task",
+ "tasks": "task",
+ "timetable": "schedule",
+ "timezone": "schedule",
+}
+# Fields dropped from the current schema that a stored payload can still carry. Every entry must
+# survive _canonicalize_payload_v1, or it classifies nothing.
+_DIFF_V1_LEGACY_DAG_FIELD_CATEGORIES: dict[str, DiffCategory] = {
+ "fail_stop": "schedule",
+ "on_failure_callback": "callback",
+ "on_success_callback": "callback",
+ "schedule": "schedule",
+}
+_RECURSIVE_MAPPING_PATHS = {
+ (),
+ ("dag",),
+ ("provenance",),
+ *_KEYED_COLLECTION_PATHS,
+}
+_DIFF_V1_TASK_GROUP_METADATA_FIELDS = frozenset(
+ {"group_display_name", "tooltip", "doc_md", "ui_color", "ui_fgcolor"}
+)
+_DIFF_V1_PUBLIC_TASK_GROUP_FIELDS = _DIFF_V1_TASK_GROUP_METADATA_FIELDS | {
+ "_group_id",
+ "prefix_group_id",
+ "children",
+ "upstream_group_ids",
+ "downstream_group_ids",
+ "upstream_task_ids",
+ "downstream_task_ids",
+ "expand_input",
+ "is_mapped",
+}
+
+
+def build_unavailable_dag_diff(
+ *,
+ base_data: dict[str, Any] | None,
+ target_data: dict[str, Any] | None,
+ reason: str,
+) -> dict[str, Any]:
+ """Report a known unavailable reason without comparing the stored payloads."""
+ return _mark_unavailable(
+ _build_diff_result(_get_schema_version(base_data), _get_schema_version(target_data)), reason
+ )
+
+
+def build_serialized_dag_diff(
+ *,
+ base_data: dict[str, Any] | None,
+ target_data: dict[str, Any] | None,
+ base_provenance: Mapping[str, Any] | None = None,
+ target_provenance: Mapping[str, Any] | None = None,
+ include_values: bool = False,
+ max_changes: int = DEFAULT_MAX_CHANGES,
+) -> dict[str, Any]:
+ """
+ Compare two serialized Dag payloads and their provenance deterministically.
+
+ Callers must authorize disclosure of the entire serialized payload, including
+ access-control roles and permissions, before setting ``include_values=True``.
+ This exposes canonicalized values, digests, and identifying path components.
+
+ ``max_changes`` limits underlying changes admitted to the result. Redacted changes
+ with the same public path and operation share a record. Repeats can be counted past
+ the limit until a change needing a new record stops the walk. When ``truncated`` is
+ true, some paths are absent and occurrence counts are lower bounds.
+
+ An ``unavailable`` result includes the reason comparison failed.
+ """
+ validate_max_changes(max_changes)
+
+ base_schema_version = _get_schema_version(base_data)
+ target_schema_version = _get_schema_version(target_data)
+ result = _build_diff_result(base_schema_version, target_schema_version)
+
+ if base_data is None or target_data is None:
+ return _mark_unavailable(result, "serialized_dag_missing")
+
+ if base_schema_version is None or target_schema_version is None:
+ return _mark_unavailable(result, "serialized_dag_schema_version_missing")
+
+ unsupported_versions = [
+ version
+ for version in (base_schema_version, target_schema_version)
+ if version not in SUPPORTED_SERIALIZED_DAG_SCHEMA_VERSIONS
+ ]
+ if unsupported_versions:
+ return _mark_unavailable(
+ result, f"unsupported_serialized_dag_schema_version:{unsupported_versions[0]}"
+ )
+
+ try:
+ base_document = _canonicalize_payload_v1(base_data)
+ target_document = _canonicalize_payload_v1(target_data)
+ base_document["provenance"] = _canonicalize_value(dict(base_provenance or {}), path=("provenance",))
+ target_document["provenance"] = _canonicalize_value(
+ dict(target_provenance or {}), path=("provenance",)
+ )
+ except (AttributeError, KeyError, OverflowError, RecursionError, TypeError, ValueError) as error:
+ log.warning(
+ "Serialized Dag diff canonicalization failed",
+ error_type=type(error).__name__,
+ base_schema_version=base_schema_version,
+ target_schema_version=target_schema_version,
+ )
+ return _mark_unavailable(result, "serialized_dag_canonicalization_failed")
+
+ collector = _ChangeCollector(max_changes=max_changes, include_values=include_values)
+ try:
+ _collect_changes(base_document, target_document, path=(), collector=collector)
+ except RecursionError:
+ log.warning(
+ "Serialized Dag diff recursion limit exceeded",
+ base_schema_version=base_schema_version,
+ target_schema_version=target_schema_version,
+ )
+ return _mark_unavailable(result, "serialized_dag_recursion_limit_exceeded")
+ except _JsonEncodingError:
+ log.warning(
+ "Serialized Dag diff JSON encoding failed",
+ base_schema_version=base_schema_version,
+ target_schema_version=target_schema_version,
+ )
+ return _mark_unavailable(result, "serialized_dag_json_encoding_failed")
+
+ result["changes"] = collector.changes
+ result["truncated"] = collector.is_truncated
+ if include_values:
+ result["values"] = {"status": "available"}
+ return result
+
+
+class _ChangeCollector:
+ def __init__(self, *, max_changes: int, include_values: bool) -> None:
+ self.changes: list[dict[str, Any]] = []
+ self.count = 0
+ self.max_changes = max_changes
+ self.include_values = include_values
+ self._truncated = False
+ self._redacted_changes: dict[tuple[tuple[str, ...], str], dict[str, Any]] = {}
+
+ @property
+ def is_truncated(self) -> bool:
+ return self._truncated
+
+ def add(
+ self,
+ *,
+ path: tuple[str, ...],
+ operation: Literal["added", "removed", "changed"],
+ before: Any,
+ after: Any,
+ ) -> None:
+ self.count += 1
+ public_path = _get_public_path(path)
+ key = (public_path, operation)
+ if not self.include_values and (existing := self._redacted_changes.get(key)) is not None:
+ # Counting a repeat of an already disclosed path adds no record and reveals no new
+ # path, so it stays exact past the bound instead of silently undercounting.
+ existing["occurrence_count"] += 1
+ return
+ if self.count > self.max_changes:
+ self._truncated = True
+ return
+
+ category = _get_category(public_path)
+ change: dict[str, Any] = {
+ "path": _format_path(path if self.include_values else public_path),
+ "operation": operation,
+ "category": category,
+ "impact": _get_impact(category),
+ "occurrence_count": 1,
+ }
+ if self.include_values:
+ change["before_digest"] = None if before is _MISSING else _get_digest(before)
+ change["after_digest"] = None if after is _MISSING else _get_digest(after)
+ if before is not _MISSING:
+ change["before_value"] = before
+ if after is not _MISSING:
+ change["after_value"] = after
+ else:
+ self._redacted_changes[key] = change
+ self.changes.append(change)
+
+
+_MISSING = object()
+
+
+def validate_max_changes(max_changes: int) -> None:
+ if not isinstance(max_changes, int) or isinstance(max_changes, bool) or max_changes < 1:
+ raise ValueError("max_changes must be a positive integer")
+ if max_changes > MAX_ALLOWED_CHANGES:
+ raise ValueError(f"max_changes must not exceed {MAX_ALLOWED_CHANGES}")
+
+
+def _get_schema_version(data: Mapping[str, Any] | None) -> int | None:
+ if not isinstance(data, Mapping):
+ return None
+ version = data.get("__version")
+ return version if isinstance(version, int) and not isinstance(version, bool) else None
+
+
+def _mark_unavailable(result: dict[str, Any], reason: str) -> dict[str, Any]:
+ result["mode"] = "unavailable"
+ result["unavailable_reason"] = reason
+ return result
+
+
+def _build_diff_result(base_schema_version: int | None, target_schema_version: int | None) -> dict[str, Any]:
+ return {
+ "diff_schema_version": DIFF_SCHEMA_VERSION,
+ "serialized_dag_schema_versions": {
+ "base": base_schema_version,
+ "target": target_schema_version,
+ },
+ "mode": "observed_state",
+ "changes": [],
+ "truncated": False,
+ # Always present so reading result["values"]["status"] is safe on every outcome.
+ "values": {"status": "unavailable"},
+ }
+
+
+def _canonicalize_payload_v1(data: dict[str, Any]) -> dict[str, Any]:
+ payload = copy.deepcopy(data)
+ version = _get_schema_version(payload)
+ if version is None:
+ raise ValueError("missing or invalid __version")
+ if version == 1:
+ DagSerialization.conversion_v1_to_v2(payload)
+ DagSerialization.conversion_v2_to_v3(payload)
+ elif version == 2:
+ DagSerialization.conversion_v2_to_v3(payload)
+ if not isinstance(payload.get("dag"), Mapping):
+ raise ValueError("missing dag object")
+ dag_defaults = {
+ field: value
+ for field, value in DagSerialization.get_schema_defaults("dag").items()
+ # Dag callback flags are enabled by their presence, even when their value is false.
+ if field not in _DAG_CALLBACK_FIELDS
+ }
+ payload["dag"] = {**dag_defaults, **payload["dag"]}
+ for field in _DAG_CALLBACK_FIELDS & payload["dag"].keys():
+ payload["dag"][field] = True
+ if "params" in payload["dag"]:
+ payload["dag"]["params"] = _normalize_params(payload["dag"]["params"])
+ _apply_task_defaults(payload)
+ payload.pop("__version", None)
+ return _canonicalize_value(payload, path=())
+
+
+@dataclass(frozen=True)
+class _TaskDefaultSets:
+ """The default sets that every task in one payload is folded against."""
+
+ schema_defaults: Mapping[str, Any]
+ outer_schema_defaults: Mapping[str, Any]
+ partial_schema_defaults: Mapping[str, Any]
+ partial_fields: frozenset[str]
+ client_task_defaults: Mapping[str, Any]
+ inherited_dates: Mapping[str, Any]
+
+
+def _build_task_default_sets(
+ payload: dict[str, Any], client_task_defaults: Mapping[str, Any]
+) -> _TaskDefaultSets:
+ # get_schema_defaults is lru_cache'd and hands back the very mapping the live Dag-hydration path
+ # reads, so every use of it in this module must stay read-only. Mutating it in place would
+ # corrupt Dag hydration process-wide, not merely produce a wrong diff.
+ schema_defaults = DagSerialization.get_schema_defaults("operator")
+ partial_fields = (
+ SerializedBaseOperator.get_serialized_fields() - SerializedMappedOperator.get_serialized_fields()
+ )
+ return _TaskDefaultSets(
+ schema_defaults=schema_defaults,
+ outer_schema_defaults={
+ field: value for field, value in schema_defaults.items() if field not in partial_fields
+ },
+ # A mapped task resolves these through partial_kwargs, so their defaults have to be applied
+ # there and encoded like any other partial value rather than left at the outer level.
+ partial_schema_defaults={
+ field: value for field, value in schema_defaults.items() if field in partial_fields
+ },
+ partial_fields=partial_fields,
+ client_task_defaults=client_task_defaults,
+ # set_task_dag_references falls back to the Dag's dates, and the serializer elides a task date
+ # that already matches, so an absent task date means the Dag's date rather than a change.
+ inherited_dates={field: payload["dag"].get(field) for field in ("start_date", "end_date")},
+ )
+
+
+def _apply_task_defaults(payload: dict[str, Any]) -> None:
+ client_defaults = payload.pop("client_defaults", None)
+ if client_defaults is None:
+ client_defaults = {}
+ if not isinstance(client_defaults, Mapping):
+ raise ValueError("client_defaults is not an object")
+
+ # Fail loudly on a section this version cannot fold in: dropping it would silently
+ # compare two payloads as equal when the unhandled defaults actually differ.
+ if unknown_sections := client_defaults.keys() - {"tasks"}:
+ raise ValueError(f"unsupported client_defaults sections: {sorted(unknown_sections)}")
+
+ client_task_defaults = client_defaults.get("tasks", {})
+ if not isinstance(client_task_defaults, Mapping):
+ raise ValueError("client_defaults.tasks is not an object")
+
+ defaults = _build_task_default_sets(payload, client_task_defaults)
+ tasks = payload["dag"].get("tasks", [])
+ if not isinstance(tasks, list):
+ raise ValueError("dag.tasks is not a list")
+ for task in tasks:
+ upgraded_task = _upgrade_task_entry(task, client_defaults)
+ template_fields = upgraded_task.get("template_fields", [])
+ if upgraded_task.get("_is_mapped"):
+ task_data = _build_mapped_task_data(upgraded_task, template_fields, defaults)
+ else:
+ task_data = _build_unmapped_task_data(upgraded_task, template_fields, defaults)
+ _apply_inherited_dates(task_data, template_fields, defaults)
+ if "params" in task_data and "params" not in template_fields:
+ task_data["params"] = _normalize_params(task_data["params"])
+ task["__var"] = task_data
+
+
+def _upgrade_task_entry(task: Any, client_defaults: Mapping[str, Any]) -> dict[str, Any]:
+ if not isinstance(task, dict) or not isinstance(task.get("__var"), Mapping):
+ raise ValueError("task entry is not an object")
+ encoded_task = OperatorSerialization._apply_defaults_to_encoded_op(
+ dict(task["__var"]), dict(client_defaults)
+ )
+ upgraded_task = OperatorSerialization._upgrade_encoded_operator(encoded_task)
+ # Operator identity is selected before client defaults are applied during hydration.
+ upgraded_task["_is_mapped"] = bool(task["__var"].get("_is_mapped", False))
+ return upgraded_task
+
+
+def _build_mapped_task_data(
+ upgraded_task: dict[str, Any], template_fields: Any, defaults: _TaskDefaultSets
+) -> dict[str, Any]:
+ task_data = {**defaults.outer_schema_defaults, **upgraded_task}
+ effective_partial_kwargs = _build_effective_partial_kwargs(task_data, template_fields, defaults)
+ task_data["partial_kwargs"] = effective_partial_kwargs
+ _normalize_retry_backoff(effective_partial_kwargs)
+ return task_data
+
+
+def _build_effective_partial_kwargs(
+ task_data: dict[str, Any], template_fields: Any, defaults: _TaskDefaultSets
+) -> dict[str, Any]:
+ """Resolve a mapped task's partial_kwargs, consuming the outer fields that move into them."""
+ partial_kwargs = task_data.get("partial_kwargs", {})
+ if not isinstance(partial_kwargs, Mapping):
+ raise ValueError("partial_kwargs is not an object")
+ # populate_operator only folds client defaults into partial_kwargs when the payload
+ # carries the key, so an absent one leaves the top-level value as the effective value.
+ effective_partial_kwargs = (
+ {
+ field: _encode_partial_field_value(field, value)
+ for field, value in defaults.client_task_defaults.items()
+ }
+ if "partial_kwargs" in task_data
+ else {}
+ )
+ for field, value in partial_kwargs.items():
+ effective_partial_kwargs[field] = (
+ value if _is_encoded_partial_value(value) else _encode_partial_field_value(field, value)
+ )
+ # Match populate_operator: partial values take precedence over outer task defaults.
+ for field in defaults.partial_fields & task_data.keys():
+ value = task_data.pop(field)
+ # Outer fields are already encoded unless template handling bypasses deserialization.
+ if field in template_fields:
+ value = _encode_json_value(value)
+ effective_partial_kwargs.setdefault(field, value)
+ for field, value in defaults.partial_schema_defaults.items():
+ effective_partial_kwargs.setdefault(field, _encode_partial_field_value(field, value))
+ return effective_partial_kwargs
+
+
+def _build_unmapped_task_data(
+ upgraded_task: dict[str, Any], template_fields: Any, defaults: _TaskDefaultSets
+) -> dict[str, Any]:
+ task_data = {**defaults.schema_defaults, **upgraded_task}
+ _normalize_retry_backoff(task_data)
+ for field in _OPERATOR_TIMEDELTA_FIELDS & task_data.keys():
+ value = task_data[field]
+ task_data[field] = (
+ _encode_json_value(value)
+ if field in template_fields and field in upgraded_task
+ else _encode_partial_field_value(field, value)
+ )
+ return task_data
+
+
+def _apply_inherited_dates(
+ task_data: dict[str, Any], template_fields: Any, defaults: _TaskDefaultSets
+) -> None:
+ for field, dag_date in defaults.inherited_dates.items():
+ if task_data.get(field) is None:
+ task_data[field] = _encode_partial_field_value(field, dag_date)
+ elif field in template_fields:
+ task_data[field] = _encode_json_value(task_data[field])
+ else:
+ task_data[field] = _encode_partial_field_value(field, task_data[field])
+
+
+# These two predicates deliberately disagree on what counts as already encoded, because the
+# serializer sites they mirror disagree: keep them apart rather than "aligning" them.
+def _is_encoded_partial_value(value: Any) -> bool:
+ # _deserialize_partial_kwargs only takes the full-deserialization path with BOTH keys present.
+ return isinstance(value, Mapping) and "__type" in value and "__var" in value
+
+
+def _is_encoded_param_attribute(value: Any) -> bool:
+ # _deserialize_param detects a legacy encoding from "__type" alone, with no "__var" required.
+ if isinstance(value, Mapping):
+ return "__type" in value
+ if isinstance(value, list):
+ return all(isinstance(item, Mapping) and "__type" in item for item in value)
+ return False
+
+
+def _normalize_params(params: Any) -> Any:
+ """Normalize legacy Params without losing the order used by the trigger form."""
+ if isinstance(params, Mapping):
+ # 2.9.2 and earlier stored params as a JSON object instead of ordered pairs.
+ pairs: Any = params.items()
+ elif isinstance(params, list):
+ pairs = params
+ else:
+ return params
+
+ normalized: dict[str, Any] = {}
+ for pair in pairs:
+ if not isinstance(pair, (list, tuple)) or len(pair) != 2:
+ # Leave an unrecognised shape to the raw comparison rather than guessing at it.
+ return params
+ name, value = pair
+ if isinstance(value, Mapping) and "__class" in value:
+ value = {
+ "default": _normalize_param_attribute(value.get("default")),
+ "description": _normalize_param_attribute(value.get("description")),
+ "schema": _normalize_param_attribute(value.get("schema") or {}),
+ "source": _normalize_param_attribute(value.get("source")),
+ }
+ else:
+ value = {
+ "default": _encode_json_value(value),
+ "description": None,
+ "schema": _encode_json_value({}),
+ "source": None,
+ }
+ normalized[str(name)] = value
+ return [[name, value] for name, value in normalized.items()]
+
+
+def _normalize_param_attribute(value: Any) -> Any:
+ # Match _deserialize_param's legacy-encoding detection without hydrating user objects.
+ if _is_encoded_param_attribute(value):
+ return value
+ return _encode_json_value(value)
+
+
+def _encode_partial_field_value(field: str, value: Any) -> Any:
+ if field in _OPERATOR_TIMEDELTA_FIELDS and value is not None:
+ return {"__type": "timedelta", "__var": value}
+ if field.endswith("_date") and value is not None and not isinstance(value, str):
+ return {"__type": "datetime", "__var": value}
+ if field == "resources":
+ # Resources serialize as a raw mapping; a plain dictionary has a dict envelope instead.
+ return value
+ return _encode_json_value(value)
+
+
+def _encode_json_value(value: Any) -> Any:
+ """Encode plain JSON without invoking object serializers."""
+ if isinstance(value, Mapping):
+ return {"__type": "dict", "__var": {key: _encode_json_value(item) for key, item in value.items()}}
+ if isinstance(value, list):
+ return [_encode_json_value(item) for item in value]
+ return value
+
+
+def _normalize_retry_backoff(task_fields: dict[str, Any]) -> None:
+ if "retry_exponential_backoff" in task_fields:
+ value = task_fields["retry_exponential_backoff"]
+ # A non-numeric stored value stays as it is so the comparison reports just this field
+ # instead of collapsing the whole diff to a canonicalization failure.
+ with suppress(TypeError, ValueError):
+ task_fields["retry_exponential_backoff"] = 2.0 if value is True else float(value)
+
+
+def _canonicalize_value(value: Any, *, path: tuple[str, ...]) -> Any:
+ if isinstance(value, Mapping):
+ if path == ("dag", "deadline"):
+ if value.get("__type") == "deadline_alert":
+ value = value["__var"]
+ value = {"name": None, **value}
+ interval = value.get("interval")
+ if isinstance(interval, (int, float)) and not isinstance(interval, bool):
+ # Diff schema v1 uses the SDK's version-2 timedelta encoding for legacy seconds.
+ value["interval"] = {
+ "__classname__": "datetime.timedelta",
+ "__version__": 2,
+ "__data__": timedelta(seconds=interval).total_seconds(),
+ }
+ return {
+ canonical_key: _canonicalize_value(item, path=path + (canonical_key,))
+ for canonical_key, item in (
+ (_canonicalize_mapping_key(key), item)
+ for key, item in sorted(value.items(), key=lambda item: _canonicalize_mapping_key(item[0]))
+ )
+ }
+ if isinstance(value, (list, tuple)):
+ canonical_values = [_canonicalize_value(item, path=path) for item in value]
+ if path == ("dag", "tasks"):
+ return _canonicalize_keyed_list(canonical_values, _get_task_id, path)
+ if path == ("dag", "dag_dependencies"):
+ return _canonicalize_keyed_list(canonical_values, _get_dependency_key, path)
+ if path in _ORDER_INSENSITIVE_LIST_PATHS:
+ return _canonicalize_keyed_list(canonical_values, _get_string_key, path)
+ return canonical_values
+ return value
+
+
+def _canonicalize_mapping_key(key: Any) -> str:
+ if isinstance(key, str) and isinstance(key, Enum):
+ return key.value
+ return str(key)
+
+
+def _canonicalize_keyed_list(
+ values: list[Any], key_getter: Callable[[Any], str], path: tuple[str, ...]
+) -> dict[str, Any]:
+ keyed_values: dict[str, Any] = {}
+ for value in values:
+ key = key_getter(value)
+ if key in keyed_values:
+ if path in _ORDER_INSENSITIVE_LIST_PATHS or (
+ path == ("dag", "dag_dependencies") and keyed_values[key] == value
+ ):
+ continue
+ raise ValueError(f"duplicate key {key!r} in /{'/'.join(path)}")
+ keyed_values[key] = value
+ return {key: keyed_values[key] for key in sorted(keyed_values)}
+
+
+def _get_task_id(task: Any) -> str:
+ if not isinstance(task, Mapping):
+ raise ValueError("task entry is not an object")
+ task_data = task.get("__var", task)
+ task_id = task_data.get("task_id") if isinstance(task_data, Mapping) else None
+ if not isinstance(task_id, str):
+ raise ValueError("task entry has no task_id")
+ return task_id
+
+
+def _get_string_key(value: Any) -> str:
+ if not isinstance(value, str):
+ raise ValueError("collection entry is not a string")
+ return value
+
+
+def _get_dependency_key(dependency: Any) -> str:
+ if not isinstance(dependency, MutableMapping):
+ raise ValueError("dependency entry is not an object")
+ # DagDependency.node_id identifies an edge by the components below alone, and ``label`` only
+ # mirrors state compared elsewhere (a task's _task_display_name, or an asset name already inside
+ # dependency_id). Keying on it turns a rename into one edge removed and another added; leaving
+ # it in the entry turns two entries on one edge into duplicates that fail canonicalization.
+ dependency.pop("label", None)
+ components = (
+ dependency.get("dependency_type"),
+ dependency.get("dependency_id"),
+ dependency.get("source"),
+ dependency.get("target"),
+ )
+ return json.dumps(components, ensure_ascii=False, separators=(",", ":"))
+
+
+def _collect_changes(
+ before: Any,
+ after: Any,
+ *,
+ path: tuple[str, ...],
+ collector: _ChangeCollector,
+) -> None:
+ if before is _MISSING and after is _MISSING:
+ return
+ if isinstance(before, Mapping) and isinstance(after, Mapping):
+ # Never branch on ``include_values`` here; see build_serialized_dag_diff.
+ if _is_task_mapping_path(path):
+ _collect_task_changes(before, after, path=path, collector=collector)
+ return
+ if _is_task_group_mapping_path(path):
+ _collect_public_field_changes(
+ before,
+ after,
+ path=path,
+ public_fields=_DIFF_V1_PUBLIC_TASK_GROUP_FIELDS,
+ collector=collector,
+ )
+ return
+ if path == _DEFAULT_ARGS_PATH:
+ _collect_default_args_changes(before, after, path=path, collector=collector)
+ return
+ if not _should_recurse_mapping(path):
+ if not _is_json_equal(before, after):
+ collector.add(path=path, operation="changed", before=before, after=after)
+ return
+ keys = sorted({str(key) for key in before} | {str(key) for key in after})
+ for key in keys:
+ before_value = before.get(key, _MISSING)
+ after_value = after.get(key, _MISSING)
+ _collect_changes(before_value, after_value, path=path + (key,), collector=collector)
+ if collector.is_truncated:
+ return
+ return
+
+ if isinstance(before, list) and isinstance(after, list):
+ if (
+ _is_task_group_child_path(path)
+ and len(before) == len(after) == 2
+ and before[0] == after[0] == "taskgroup"
+ and isinstance(before[1], Mapping)
+ and isinstance(after[1], Mapping)
+ ):
+ _collect_changes(before[1], after[1], path=path + ("1",), collector=collector)
+ return
+ if not _is_json_equal(before, after):
+ collector.add(path=path, operation="changed", before=before, after=after)
+ return
+
+ if before is _MISSING:
+ collector.add(path=path, operation="added", before=_MISSING, after=after)
+ elif after is _MISSING:
+ collector.add(path=path, operation="removed", before=before, after=_MISSING)
+ elif not _is_json_equal(before, after):
+ collector.add(path=path, operation="changed", before=before, after=after)
+
+
+def _collect_task_changes(
+ before: Mapping[str, Any],
+ after: Mapping[str, Any],
+ *,
+ path: tuple[str, ...],
+ collector: _ChangeCollector,
+) -> None:
+ if len(path) == 3:
+ _collect_changes(
+ before.get("__type", _MISSING),
+ after.get("__type", _MISSING),
+ path=path + ("__type",),
+ collector=collector,
+ )
+ if collector.is_truncated:
+ return
+ before, after = before["__var"], after["__var"]
+ public_fields = _DIFF_V1_PUBLIC_TASK_FIELDS - {"__type"}
+ else:
+ public_fields = _DIFF_V1_PUBLIC_PARTIAL_TASK_FIELDS
+ _collect_public_field_changes(before, after, path=path, public_fields=public_fields, collector=collector)
+
+
+def _collect_default_args_changes(
+ before: Mapping[str, Any],
+ after: Mapping[str, Any],
+ *,
+ path: tuple[str, ...],
+ collector: _ChangeCollector,
+) -> None:
+ """Compare a Dag's default_args key by key so each key carries its own impact."""
+ if not (_is_encoded_default_args(before) and _is_encoded_default_args(after)):
+ # An envelope this version cannot read stays one opaque leaf: the allowlist was written for
+ # the keys of a dict envelope, so applying it to another shape could name something else.
+ if not _is_json_equal(before, after):
+ collector.add(path=path, operation="changed", before=before, after=after)
+ return
+ _collect_public_field_changes(
+ before["__var"],
+ after["__var"],
+ path=path,
+ public_fields=_DIFF_V1_PUBLIC_DEFAULT_ARGS_FIELDS,
+ collector=collector,
+ )
+
+
+def _is_encoded_default_args(value: Any) -> bool:
+ return (
+ isinstance(value, Mapping)
+ and value.get("__type") == "dict"
+ and isinstance(value.get("__var"), Mapping)
+ )
+
+
+def _collect_public_field_changes(
+ before: Mapping[str, Any],
+ after: Mapping[str, Any],
+ *,
+ path: tuple[str, ...],
+ public_fields: frozenset[str] | set[str],
+ collector: _ChangeCollector,
+) -> None:
+ """Walk allowlisted fields one by one and report everything else as a single change."""
+ keys = {str(key) for key in before} | {str(key) for key in after}
+ for key in sorted(keys & public_fields):
+ _collect_changes(
+ before.get(key, _MISSING),
+ after.get(key, _MISSING),
+ path=path + (key,),
+ collector=collector,
+ )
+ if collector.is_truncated:
+ return
+
+ before_custom_fields = {key: before[key] for key in before if key not in public_fields}
+ after_custom_fields = {key: after[key] for key in after if key not in public_fields}
+ if not _is_json_equal(before_custom_fields, after_custom_fields):
+ collector.add(
+ path=path + (_CUSTOM_TASK_FIELDS_PATH_COMPONENT,),
+ operation="changed",
+ before=before_custom_fields,
+ after=after_custom_fields,
+ )
+
+
+def _is_task_mapping_path(path: tuple[str, ...]) -> bool:
+ return path[:2] == ("dag", "tasks") and (
+ len(path) == 3 or (len(path) == 4 and path[3] == "partial_kwargs")
+ )
+
+
+class _TaskGroupPath(NamedTuple):
+ """One reading of the task group path grammar, shared by everything that walks it."""
+
+ field_index: int
+ id_positions: tuple[int, ...]
+ remainder: tuple[str, ...]
+
+
+def _classify_task_group_path(path: tuple[str, ...]) -> _TaskGroupPath | None:
+ """
+ Split a task group path into its nesting prefix and what follows the innermost group.
+
+ The grammar is ``dag/task_group ( /children/<group_id>/1 )* [ /children/<child_id> | /<field> ... ]``:
+ a child entry is the two-element ``["taskgroup", {...}]`` list the serializer writes, so the
+ literal ``1`` selects the nested group mapping inside it. ``field_index`` is where the innermost
+ group's own keys start, ``id_positions`` the group-id components a redacted path must mask, and
+ ``remainder`` the components from ``field_index`` on: ``()`` for the group mapping itself,
+ ``("children",)`` for its children mapping, and a two-element ``("children", child_id)`` for one
+ child entry. Returns ``None`` for a path that is not rooted at the Dag's task group.
+ """
+ if path[:2] != ("dag", "task_group"):
+ return None
+ index = 2
+ id_positions: list[int] = []
+ while len(path) >= index + 2 and path[index] == "children":
+ id_positions.append(index + 1)
+ if len(path) < index + 3 or path[index + 2] != "1":
+ break
+ index += 3
+ return _TaskGroupPath(index, tuple(id_positions), path[index:])
+
+
+def _is_task_group_mapping_path(path: tuple[str, ...]) -> bool:
+ task_group_path = _classify_task_group_path(path)
+ return task_group_path is not None and task_group_path.remainder == ()
+
+
+def _should_recurse_mapping(path: tuple[str, ...]) -> bool:
+ if path in _RECURSIVE_MAPPING_PATHS:
+ return True
+ task_group_path = _classify_task_group_path(path)
+ return task_group_path is not None and task_group_path.remainder == ("children",)
+
+
+def _is_task_group_child_path(path: tuple[str, ...]) -> bool:
+ task_group_path = _classify_task_group_path(path)
+ return (
+ task_group_path is not None
+ and len(task_group_path.remainder) == 2
+ and task_group_path.remainder[0] == "children"
+ )
+
+
+def _get_public_path(path: tuple[str, ...]) -> tuple[str, ...]:
+ if path and path[0] not in _DIFF_V1_PUBLIC_ROOT_FIELDS:
+ # Root is the one level no allowlisted walk covers, so an unrecognised section name would
+ # otherwise be the single path component a redacted caller receives verbatim.
+ return (_CUSTOM_TASK_FIELDS_PATH_COMPONENT,)
+ if (task_group_path := _classify_task_group_path(path)) is not None:
+ public_path = list(path)
+ for position in task_group_path.id_positions:
+ public_path[position] = "*"
+ remainder = task_group_path.remainder
+ if remainder and remainder[0] not in _DIFF_V1_PUBLIC_TASK_GROUP_FIELDS:
+ # Unreachable by construction (_collect_public_field_changes aggregates non-public group
+ # keys first), but truncating like the task branch stops a future caller leaking a key.
+ return (*public_path[: task_group_path.field_index], _CUSTOM_TASK_FIELDS_PATH_COMPONENT)
+ return tuple(public_path)
+ if len(path) >= 3 and path[:2] in _KEYED_COLLECTION_PATHS:
+ path = (*path[:2], "*", *path[3:])
+ if len(path) >= 4 and path[:2] == ("dag", "tasks"):
+ field_index = 4 if path[3] == "partial_kwargs" and len(path) >= 5 else 3
+ public_fields = (
+ _DIFF_V1_PUBLIC_PARTIAL_TASK_FIELDS if field_index == 4 else _DIFF_V1_PUBLIC_TASK_FIELDS
+ )
+ if path[field_index] not in public_fields:
+ return (*path[:field_index], _CUSTOM_TASK_FIELDS_PATH_COMPONENT)
+ return path
+
+
+def _format_path(path: tuple[str, ...]) -> str:
+ return "/" + "/".join(component.replace("~", "~0").replace("/", "~1") for component in path)
+
+
+class _JsonEncodingError(TypeError):
+ """Distinguish unsupported JSON values from comparison implementation errors."""
+
+
+def _serialize_canonical_json(value: Any) -> str:
+ try:
+ return json.dumps(value, ensure_ascii=False, separators=(",", ":"), sort_keys=True)
+ except TypeError as error:
+ raise _JsonEncodingError from error
+
+
+def _is_json_equal(before: Any, after: Any) -> bool:
+ # Compare canonical JSON, not Python ==, so a JSON type change (1 vs 1.0, True vs 1,
+ # False vs 0) registers as a change instead of collapsing under Python equality.
+ return _serialize_canonical_json(before) == _serialize_canonical_json(after)
+
+
+def _get_digest(value: Any) -> str:
+ # SHA-256 provides a content fingerprint, not password protection.
+ # Stored payloads can carry escaped lone surrogates, which strict UTF-8 refuses to encode;
+ # surrogatepass keeps them digestible without altering the bytes of any other payload.
+ canonical_json = _serialize_canonical_json(value).encode("utf-8", errors="surrogatepass")
+ return f"sha256:{hashlib.sha256(canonical_json).hexdigest()}"
Review Comment:
SHA-256 provides deterministic content fingerprints here; it is not used for password storage or verification.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add serialized Dag version diff engine [airflow]
Posted by "kaxil (via GitHub)" <gi...@apache.org>.
kaxil commented on code in PR #69864:
URL: https://github.com/apache/airflow/pull/69864#discussion_r4032039057
##########
airflow-core/src/airflow/serialization/dag_version_diff.py:
##########
@@ -0,0 +1,1281 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""
+Observed-state diffs for serialized Dag payloads.
+
+Diff schema v1 -- the wire format every entry point here returns.
+:func:`build_serialized_dag_diff` and :func:`build_unavailable_dag_diff` both return one
+dictionary in the shape described below, and ``DIFF_SCHEMA_VERSION`` is the version of that
+shape: bump it whenever a client-observable part of this contract changes.
+
+Top-level keys, all always present:
+
+* ``diff_schema_version`` -- ``int``, the value of ``DIFF_SCHEMA_VERSION``.
+* ``serialized_dag_schema_versions`` -- ``{"base": int | None, "target": int | None}``, the
+ ``__version`` each stored payload carried. ``None`` means the key was absent or not an
+ integer; ``bool`` is rejected, so ``__version: true`` reads as ``None`` and never as
+ version 1. A version in ``SUPPORTED_SERIALIZED_DAG_SCHEMA_VERSIONS`` is upgraded to the
+ newest supported version before anything is compared.
+* ``mode`` -- ``"observed_state"`` when the two payloads were compared, ``"unavailable"``
+ when no comparison happened.
+* ``changes`` -- the change records described below, ordered by the deterministic walk.
+ Always empty when ``mode`` is ``"unavailable"``.
+* ``truncated`` -- ``True`` when more underlying changes exist than ``max_changes`` allowed.
+ Always ``False`` when ``mode`` is ``"unavailable"``.
+* ``values`` -- ``{"status": "available" | "unavailable"}``. ``"available"`` only for an
+ ``"observed_state"`` result whose caller authorized value disclosure, so
+ ``result["values"]["status"]`` is safe to read on every result of every entry point.
+
+``unavailable_reason`` is the one conditional key: present exactly when ``mode`` is
+``"unavailable"``. :func:`build_serialized_dag_diff` produces:
+
+* ``serialized_dag_missing`` -- one side has no stored payload.
+* ``serialized_dag_schema_version_missing`` -- one side carries no usable ``__version``.
+* ``unsupported_serialized_dag_schema_version:<n>`` -- ``<n>`` is the first unsupported
+ version found, base before target.
+* ``serialized_dag_canonicalization_failed`` -- a payload could not be normalized (malformed
+ structure, an unsupported ``client_defaults`` section, an unencodable value).
+* ``serialized_dag_recursion_limit_exceeded`` -- the comparison walk ran too deep.
+* ``serialized_dag_json_encoding_failed`` -- a value could not be encoded as canonical JSON.
+
+:meth:`~airflow.models.dag_version.DagVersion.get_diff` reuses
+``serialized_dag_decode_failed``, for a stored payload that will not decompress or parse, and
+``deadline_alert_missing``, for a payload that references a deadline alert row that is gone.
+:func:`build_unavailable_dag_diff` passes its caller's reason through unchanged, so a new
+caller extends this list rather than inventing an undocumented value.
+
+Each change record carries these keys in both disclosure modes:
+
+* ``path`` -- a JSON-Pointer-style path into a synthetic document with two roots: ``/dag``
+ (the canonicalized serialized Dag, with ``__version`` dropped and client defaults folded
+ into the tasks) and ``/provenance`` (the provenance mapping the caller supplied --
+ ``bundle_name``, ``bundle_version`` and ``version_data`` from ``get_diff``). ``~`` and
+ ``/`` inside a component are escaped as ``~0`` and ``~1``.
+* ``operation`` -- ``"added"``, ``"removed"`` or ``"changed"``.
+* ``category`` -- a :data:`DiffCategory`: ``task`` for task definitions and the Dag fields
+ that shape task execution, ``dependency`` for ``dag_dependencies`` and downstream task
+ ids, ``schedule`` for timetable, dates and concurrency, ``param`` for params,
+ ``asset`` for task inlets and outlets, ``deadline`` for deadline alerts,
+ ``callback`` for callback presence, ``metadata`` for descriptive and display fields,
+ ``authorization`` for ``access_control``, ``provenance`` for everything under
+ ``/provenance`` plus the file and bundle locators, and ``unknown`` for anything
+ unclassified. ``_DIFF_V1_DAG_FIELD_CATEGORIES`` is the authoritative Dag-field mapping;
+ an unclassified task field falls back to ``task``, which over-reports impact.
+ ``default_args`` is the one Dag field classified per key rather than as a whole, because its
+ keys are the operator fields they will be applied to: an allowlisted key takes that field's
+ category (``owner`` is ``metadata``, ``retries`` is ``task``), and the aggregate record for
+ the rest keeps the conservative ``task``. Only a stored value that is not a readable dict
+ envelope classifies as ``task`` as a whole.
+* ``impact`` -- a :data:`DiffImpact` derived from ``category``: ``provenance``, ``metadata``
+ and ``authorization`` carry through unchanged, every operational category becomes
+ ``execution``, and ``unknown`` stays ``unknown``.
+* ``occurrence_count`` -- how many underlying changes the record stands for.
+
+A caller that authorized values (``include_values=True``, ``values.status`` ``"available"``)
+additionally gets:
+
+* ``before_digest`` / ``after_digest`` -- ``"sha256:<hex>"`` over the canonical JSON of that
+ side, or ``None`` when that side is missing. Both keys are always present.
+* ``before_value`` / ``after_value`` -- the canonicalized value, which is the stored one after
+ schema and client defaults are folded in and params are normalized. ``before_value`` is absent from
+ an ``added`` record and ``after_value`` from a ``removed`` one, which is how a missing side
+ is told apart from a stored ``null``.
+
+``path`` and ``occurrence_count`` also differ between the modes. A redacted record reports the
+public path: members of keyed collections (``/dag/tasks``, ``/dag/dag_dependencies``,
+``/dag/tags``, ``/dag/allowed_run_types`` and task group children) collapse to ``*`` instead
+of naming a task, tag or group, and records sharing a public path and operation are merged
+into one whose ``occurrence_count`` counts the merged changes. An authorized record keeps the
+identifying component and always has an ``occurrence_count`` of 1. Fields outside the v1 task,
+task group and ``default_args`` allowlists are aggregated into a single ``custom_fields`` record
+in both modes, so neither a private serializer field nor a user-chosen ``default_args`` key is
+ever named by either -- ``/dag/default_args`` names only allowlisted operator field names, which
+is why its keys can be classified at all. A top-level key outside the v1 root allowlist is
+likewise reported as ``/custom_fields`` when redacted, while an authorized record keeps its real
+root path.
+
+``max_changes`` bounds the result rather than the traversal: both modes walk the same shape and
+reach the same changes, so authorization never decides which changes exist, and the public paths
+either mode reports -- and the order of their first occurrence -- are the same. What the bound
+admits does differ. An authorized record stands for one change, so the bound caps those directly.
+A redacted record stands for every change sharing its public path and operation, and a change
+reaching a record already in the result costs no new one, so redacted counting continues past the
+bound until a change needs a record the bound will not allow. The same payload at the same bound
+can therefore report more occurrences, and a different ``truncated``, when values are withheld.
+
+``truncated`` reports that such a change was dropped. When it is ``False`` every changed path
+is present with an exact ``occurrence_count``. When it is ``True`` the walk stopped early: some
+paths are absent rather than unchanged, and every ``occurrence_count`` is a lower bound, since
+occurrences after the stop were never reached.
+
+Typed dictionaries for the result and the change record are deliberately deferred -- the
+record shape varies with disclosure mode, so modelling it belongs with the REST layer that
+exposes it. :data:`DiffCategory` and :data:`DiffImpact` are stable enough to generate enums
+from today.
+"""
+
+from __future__ import annotations
+
+import copy
+import hashlib
+import json
+from collections.abc import Callable, Mapping, MutableMapping
+from contextlib import suppress
+from dataclasses import dataclass
+from datetime import timedelta
+from enum import Enum
+from typing import Any, Literal, NamedTuple
+
+import structlog
+
+from airflow.serialization.definitions.baseoperator import SerializedBaseOperator
+from airflow.serialization.definitions.mappedoperator import SerializedMappedOperator
+from airflow.serialization.serialized_objects import (
+ _DAG_CALLBACK_FIELDS,
+ _OPERATOR_TIMEDELTA_FIELDS,
+ DagSerialization,
+ OperatorSerialization,
+)
+
+log = structlog.get_logger(__name__)
+
+DIFF_SCHEMA_VERSION = 1
+DEFAULT_MAX_CHANGES = 500
+MAX_ALLOWED_CHANGES = 5000
+SUPPORTED_SERIALIZED_DAG_SCHEMA_VERSIONS = frozenset((1, 2, 3))
+
+DiffCategory = Literal[
+ "asset",
+ "authorization",
+ "callback",
+ "deadline",
+ "dependency",
+ "metadata",
+ "param",
+ "provenance",
+ "schedule",
+ "task",
+ "unknown",
+]
+DiffImpact = Literal["authorization", "execution", "metadata", "provenance", "unknown"]
+
+_ORDER_INSENSITIVE_LIST_PATHS = {
+ ("dag", "tags"),
+ ("dag", "allowed_run_types"),
+}
+_KEYED_COLLECTION_PATHS = {
+ ("dag", "tasks"),
+ ("dag", "dag_dependencies"),
+ *_ORDER_INSENSITIVE_LIST_PATHS,
+}
+# Canonicalization folds away __version and client_defaults and adds provenance, so these are the
+# only top-level sections the walk is allowed to name.
+_DIFF_V1_PUBLIC_ROOT_FIELDS = frozenset({"dag", "provenance"})
+_CUSTOM_TASK_FIELDS_PATH_COMPONENT = "custom_fields"
+# This allowlist is part of diff schema v1. Serializer schema changes must not
+# silently change the paths visible to callers of the diff API.
+_DIFF_V1_PUBLIC_TASK_FIELDS = frozenset(
+ {
+ "__type",
+ "_can_skip_downstream",
+ "_disallow_kwargs_override",
+ "_expand_input_attr",
+ "_is_empty",
+ "_is_mapped",
+ "_is_sensor",
+ "_logger_name",
+ "_needs_expansion",
+ "_operator_extra_links",
+ "_operator_name",
+ "_task_display_name",
+ "_task_module",
+ "allow_nested_operators",
+ "depends_on_past",
+ "do_xcom_push",
+ "doc",
+ "doc_json",
+ "doc_md",
+ "doc_rst",
+ "doc_yaml",
+ "downstream_task_ids",
+ "email",
+ "email_on_failure",
+ "email_on_retry",
+ "end_date",
+ "execution_timeout",
+ "executor",
+ "executor_config",
+ "expand_input",
+ "has_on_execute_callback",
+ "has_on_failure_callback",
+ "has_on_retry_callback",
+ "has_on_skipped_callback",
+ "has_on_success_callback",
+ "has_retry_policy",
+ "ignore_first_depends_on_past",
+ "inlets",
+ "is_setup",
+ "is_stub",
+ "is_teardown",
+ "map_index_template",
+ "max_active_tis_per_dag",
+ "max_active_tis_per_dagrun",
+ "max_retry_delay",
+ "multiple_outputs",
+ "on_failure_fail_dagrun",
+ "op_kwargs_expand_input",
+ "outlets",
+ "owner",
+ "params",
+ "partial_kwargs",
+ "pool",
+ "pool_slots",
+ "priority_weight",
+ "python_callable_name",
+ "queue",
+ "render_template_as_native_obj",
+ "reschedule",
+ "resources",
+ "retries",
+ "retry_delay",
+ "retry_exponential_backoff",
+ "run_as_user",
+ "start_date",
+ "start_from_trigger",
+ "start_trigger_args",
+ "task_id",
+ "task_type",
+ "template_ext",
+ "template_fields",
+ "template_fields_renderers",
+ "trigger_rule",
+ "ui_color",
+ "ui_fgcolor",
+ "wait_for_downstream",
+ "wait_for_past_depends_before_skipping",
+ "weight_rule",
+ }
+)
+_DIFF_V1_REDACTED_SCHEMA_TASK_FIELDS = frozenset({"_arg_bindings"})
+# Task-field categories used by _get_category.
+_DIFF_V1_TASK_ASSET_FIELDS = frozenset({"inlets", "outlets"})
+_DIFF_V1_TASK_PARAM_FIELDS = frozenset({"params"})
+_DIFF_V1_TASK_DEPENDENCY_FIELDS = frozenset({"downstream_task_ids"})
+_DIFF_V1_TASK_METADATA_FIELDS = frozenset(
+ {
+ "_operator_name",
+ "doc",
+ "doc_json",
+ "doc_md",
+ "doc_rst",
+ "doc_yaml",
+ "owner",
+ "ui_color",
+ "ui_fgcolor",
+ "_task_display_name",
+ "task_display_name",
+ }
+)
+_DIFF_V1_PUBLIC_PARTIAL_TASK_FIELDS = _DIFF_V1_PUBLIC_TASK_FIELDS | {"task_display_name"}
+_DEFAULT_ARGS_PATH = ("dag", "default_args")
+_RETRY_BACKOFF_FIELD = "retry_exponential_backoff"
+# Dag fields the schema gives no default for, added after the first serializer versions.
+_ABSENT_AS_NULL_DAG_FIELDS = ("allowed_run_types", "deadline")
+# These hydrate to sets (_deserialize_operator_field for a task, TaskGroup.*_ids.update for a group),
+# so a reordered stored list means the same edges and must not read as a dependency change.
+_SET_VALUED_ID_FIELDS = frozenset(
+ {"downstream_task_ids", "upstream_task_ids", "downstream_group_ids", "upstream_group_ids"}
+)
+# A Dag's default_args holds the same operator __init__ kwargs partial_kwargs holds -- the
+# serializer rewrites _HAS_FLAG_FIELDS in both -- so it reuses that allowlist. "__type" names the
+# stored value's own dict envelope rather than a key inside it.
+_DIFF_V1_PUBLIC_DEFAULT_ARGS_FIELDS = _DIFF_V1_PUBLIC_PARTIAL_TASK_FIELDS - {"__type"}
+# Classify every Dag schema field explicitly so new fields require a policy decision.
+_DIFF_V1_DAG_FIELD_CATEGORIES: dict[str, DiffCategory] = {
+ "_concurrency": "schedule",
+ "_processor_dags_folder": "provenance",
+ "access_control": "authorization",
+ "allowed_run_types": "schedule",
+ "bundle_name": "provenance",
+ "catchup": "schedule",
+ "dag_dependencies": "dependency",
+ "dag_display_name": "metadata",
+ "dag_id": "metadata",
+ "dagrun_timeout": "schedule",
+ "deadline": "deadline",
+ # Only reached when the stored value is not the dict envelope _collect_default_args_changes
+ # walks per key; an unreadable envelope keeps the conservative impact of its widest key.
+ "default_args": "task",
+ "description": "metadata",
+ # Decides whether a run pins a bundle version, so it changes which code later runs execute
+ # and whether triggering a historical version is allowed at all.
+ "disable_bundle_versioning": "task",
+ "doc_md": "metadata",
+ "edge_info": "metadata",
+ "end_date": "schedule",
+ "fail_fast": "schedule",
+ "fileloc": "provenance",
+ "has_on_failure_callback": "callback",
+ "has_on_success_callback": "callback",
+ "is_paused_upon_creation": "schedule",
+ "max_active_runs": "schedule",
+ "max_active_tasks": "schedule",
+ "max_consecutive_failed_dag_runs": "schedule",
+ "owner_links": "metadata",
+ "params": "param",
+ "relative_fileloc": "provenance",
+ "render_template_as_native_obj": "task",
+ "rerun_with_latest_version": "task",
+ "start_date": "schedule",
+ "tags": "metadata",
+ "task_group": "task",
+ "tasks": "task",
+ "timetable": "schedule",
+ "timezone": "schedule",
+}
+# Fields dropped from the current schema that a stored payload can still carry. Every entry must
+# survive _canonicalize_payload_v1, or it classifies nothing.
+_DIFF_V1_LEGACY_DAG_FIELD_CATEGORIES: dict[str, DiffCategory] = {
+ "fail_stop": "schedule",
+ "on_failure_callback": "callback",
+ "on_success_callback": "callback",
+ "schedule": "schedule",
+}
+_RECURSIVE_MAPPING_PATHS = {
+ (),
+ ("dag",),
+ ("provenance",),
+ *_KEYED_COLLECTION_PATHS,
+}
+_DIFF_V1_TASK_GROUP_METADATA_FIELDS = frozenset(
+ {"group_display_name", "tooltip", "doc_md", "ui_color", "ui_fgcolor"}
+)
+_DIFF_V1_PUBLIC_TASK_GROUP_FIELDS = _DIFF_V1_TASK_GROUP_METADATA_FIELDS | {
+ "_group_id",
+ "prefix_group_id",
+ "children",
+ "upstream_group_ids",
+ "downstream_group_ids",
+ "upstream_task_ids",
+ "downstream_task_ids",
+ "expand_input",
+ "is_mapped",
+}
+
+
+def build_unavailable_dag_diff(
+ *,
+ base_data: dict[str, Any] | None,
+ target_data: dict[str, Any] | None,
+ reason: str,
+) -> dict[str, Any]:
+ """Report a known unavailable reason without comparing the stored payloads."""
+ return _mark_unavailable(
+ _build_diff_result(_get_schema_version(base_data), _get_schema_version(target_data)), reason
+ )
+
+
+def build_serialized_dag_diff(
+ *,
+ base_data: dict[str, Any] | None,
+ target_data: dict[str, Any] | None,
+ base_provenance: Mapping[str, Any] | None = None,
+ target_provenance: Mapping[str, Any] | None = None,
+ include_values: bool = False,
+ max_changes: int = DEFAULT_MAX_CHANGES,
+) -> dict[str, Any]:
+ """
+ Compare two serialized Dag payloads and their provenance deterministically.
+
+ Callers must authorize disclosure of the entire serialized payload, including
+ access-control roles and permissions, before setting ``include_values=True``.
+ This exposes canonicalized values, digests, and identifying path components.
+
+ ``max_changes`` limits underlying changes admitted to the result. Redacted changes
+ with the same public path and operation share a record. Repeats can be counted past
+ the limit until a change needing a new record stops the walk. When ``truncated`` is
+ true, some paths are absent and occurrence counts are lower bounds.
+
+ An ``unavailable`` result includes the reason comparison failed.
+ """
+ validate_max_changes(max_changes)
+
+ base_schema_version = _get_schema_version(base_data)
+ target_schema_version = _get_schema_version(target_data)
+ result = _build_diff_result(base_schema_version, target_schema_version)
+
+ if base_data is None or target_data is None:
+ return _mark_unavailable(result, "serialized_dag_missing")
+
+ if base_schema_version is None or target_schema_version is None:
+ return _mark_unavailable(result, "serialized_dag_schema_version_missing")
+
+ unsupported_versions = [
+ version
+ for version in (base_schema_version, target_schema_version)
+ if version not in SUPPORTED_SERIALIZED_DAG_SCHEMA_VERSIONS
+ ]
+ if unsupported_versions:
+ return _mark_unavailable(
+ result, f"unsupported_serialized_dag_schema_version:{unsupported_versions[0]}"
+ )
+
+ try:
+ base_document = _canonicalize_payload_v1(base_data)
+ target_document = _canonicalize_payload_v1(target_data)
+ base_document["provenance"] = _canonicalize_value(dict(base_provenance or {}), path=("provenance",))
+ target_document["provenance"] = _canonicalize_value(
+ dict(target_provenance or {}), path=("provenance",)
+ )
+ except (AttributeError, KeyError, OverflowError, RecursionError, TypeError, ValueError) as error:
+ log.warning(
+ "Serialized Dag diff canonicalization failed",
+ error_type=type(error).__name__,
Review Comment:
This logs the exception class and nothing else, so the messages added to make these failures loud do not survive. `_apply_task_defaults` raises `ValueError(f"unsupported client_defaults sections: {sorted(unknown_sections)}")` for exactly the case from two rounds ago, and `dag.tasks is not a list`, `task entry has no task_id` and `duplicate key ... in /dag/tasks` all land in this same handler. What an operator sees in every one of those cases is `Serialized Dag diff canonicalization failed base_schema_version=3 error_type=ValueError target_schema_version=3`.
The tuple also catches `AttributeError`, `KeyError` and `TypeError`, so an engine bug arrives here too and reads identically to a new serializer section. `exc_info=error`, or a `reason=str(error)` field, tells them apart.
##########
airflow-core/src/airflow/serialization/dag_version_diff.py:
##########
@@ -0,0 +1,1281 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""
+Observed-state diffs for serialized Dag payloads.
+
+Diff schema v1 -- the wire format every entry point here returns.
+:func:`build_serialized_dag_diff` and :func:`build_unavailable_dag_diff` both return one
+dictionary in the shape described below, and ``DIFF_SCHEMA_VERSION`` is the version of that
+shape: bump it whenever a client-observable part of this contract changes.
+
+Top-level keys, all always present:
+
+* ``diff_schema_version`` -- ``int``, the value of ``DIFF_SCHEMA_VERSION``.
+* ``serialized_dag_schema_versions`` -- ``{"base": int | None, "target": int | None}``, the
+ ``__version`` each stored payload carried. ``None`` means the key was absent or not an
+ integer; ``bool`` is rejected, so ``__version: true`` reads as ``None`` and never as
+ version 1. A version in ``SUPPORTED_SERIALIZED_DAG_SCHEMA_VERSIONS`` is upgraded to the
+ newest supported version before anything is compared.
+* ``mode`` -- ``"observed_state"`` when the two payloads were compared, ``"unavailable"``
+ when no comparison happened.
+* ``changes`` -- the change records described below, ordered by the deterministic walk.
+ Always empty when ``mode`` is ``"unavailable"``.
+* ``truncated`` -- ``True`` when more underlying changes exist than ``max_changes`` allowed.
+ Always ``False`` when ``mode`` is ``"unavailable"``.
+* ``values`` -- ``{"status": "available" | "unavailable"}``. ``"available"`` only for an
+ ``"observed_state"`` result whose caller authorized value disclosure, so
+ ``result["values"]["status"]`` is safe to read on every result of every entry point.
+
+``unavailable_reason`` is the one conditional key: present exactly when ``mode`` is
+``"unavailable"``. :func:`build_serialized_dag_diff` produces:
+
+* ``serialized_dag_missing`` -- one side has no stored payload.
+* ``serialized_dag_schema_version_missing`` -- one side carries no usable ``__version``.
+* ``unsupported_serialized_dag_schema_version:<n>`` -- ``<n>`` is the first unsupported
+ version found, base before target.
+* ``serialized_dag_canonicalization_failed`` -- a payload could not be normalized (malformed
+ structure, an unsupported ``client_defaults`` section, an unencodable value).
+* ``serialized_dag_recursion_limit_exceeded`` -- the comparison walk ran too deep.
+* ``serialized_dag_json_encoding_failed`` -- a value could not be encoded as canonical JSON.
+
+:meth:`~airflow.models.dag_version.DagVersion.get_diff` reuses
+``serialized_dag_decode_failed``, for a stored payload that will not decompress or parse, and
+``deadline_alert_missing``, for a payload that references a deadline alert row that is gone.
+:func:`build_unavailable_dag_diff` passes its caller's reason through unchanged, so a new
+caller extends this list rather than inventing an undocumented value.
+
+Each change record carries these keys in both disclosure modes:
+
+* ``path`` -- a JSON-Pointer-style path into a synthetic document with two roots: ``/dag``
+ (the canonicalized serialized Dag, with ``__version`` dropped and client defaults folded
+ into the tasks) and ``/provenance`` (the provenance mapping the caller supplied --
+ ``bundle_name``, ``bundle_version`` and ``version_data`` from ``get_diff``). ``~`` and
+ ``/`` inside a component are escaped as ``~0`` and ``~1``.
+* ``operation`` -- ``"added"``, ``"removed"`` or ``"changed"``.
+* ``category`` -- a :data:`DiffCategory`: ``task`` for task definitions and the Dag fields
+ that shape task execution, ``dependency`` for ``dag_dependencies`` and downstream task
+ ids, ``schedule`` for timetable, dates and concurrency, ``param`` for params,
+ ``asset`` for task inlets and outlets, ``deadline`` for deadline alerts,
+ ``callback`` for callback presence, ``metadata`` for descriptive and display fields,
+ ``authorization`` for ``access_control``, ``provenance`` for everything under
+ ``/provenance`` plus the file and bundle locators, and ``unknown`` for anything
+ unclassified. ``_DIFF_V1_DAG_FIELD_CATEGORIES`` is the authoritative Dag-field mapping;
+ an unclassified task field falls back to ``task``, which over-reports impact.
+ ``default_args`` is the one Dag field classified per key rather than as a whole, because its
+ keys are the operator fields they will be applied to: an allowlisted key takes that field's
+ category (``owner`` is ``metadata``, ``retries`` is ``task``), and the aggregate record for
+ the rest keeps the conservative ``task``. Only a stored value that is not a readable dict
+ envelope classifies as ``task`` as a whole.
+* ``impact`` -- a :data:`DiffImpact` derived from ``category``: ``provenance``, ``metadata``
+ and ``authorization`` carry through unchanged, every operational category becomes
+ ``execution``, and ``unknown`` stays ``unknown``.
+* ``occurrence_count`` -- how many underlying changes the record stands for.
+
+A caller that authorized values (``include_values=True``, ``values.status`` ``"available"``)
+additionally gets:
+
+* ``before_digest`` / ``after_digest`` -- ``"sha256:<hex>"`` over the canonical JSON of that
+ side, or ``None`` when that side is missing. Both keys are always present.
+* ``before_value`` / ``after_value`` -- the canonicalized value, which is the stored one after
+ schema and client defaults are folded in and params are normalized. ``before_value`` is absent from
+ an ``added`` record and ``after_value`` from a ``removed`` one, which is how a missing side
+ is told apart from a stored ``null``.
+
+``path`` and ``occurrence_count`` also differ between the modes. A redacted record reports the
+public path: members of keyed collections (``/dag/tasks``, ``/dag/dag_dependencies``,
+``/dag/tags``, ``/dag/allowed_run_types`` and task group children) collapse to ``*`` instead
+of naming a task, tag or group, and records sharing a public path and operation are merged
+into one whose ``occurrence_count`` counts the merged changes. An authorized record keeps the
+identifying component and always has an ``occurrence_count`` of 1. Fields outside the v1 task,
+task group and ``default_args`` allowlists are aggregated into a single ``custom_fields`` record
+in both modes, so neither a private serializer field nor a user-chosen ``default_args`` key is
+ever named by either -- ``/dag/default_args`` names only allowlisted operator field names, which
+is why its keys can be classified at all. A top-level key outside the v1 root allowlist is
+likewise reported as ``/custom_fields`` when redacted, while an authorized record keeps its real
+root path.
+
+``max_changes`` bounds the result rather than the traversal: both modes walk the same shape and
+reach the same changes, so authorization never decides which changes exist, and the public paths
+either mode reports -- and the order of their first occurrence -- are the same. What the bound
+admits does differ. An authorized record stands for one change, so the bound caps those directly.
+A redacted record stands for every change sharing its public path and operation, and a change
+reaching a record already in the result costs no new one, so redacted counting continues past the
+bound until a change needs a record the bound will not allow. The same payload at the same bound
+can therefore report more occurrences, and a different ``truncated``, when values are withheld.
+
+``truncated`` reports that such a change was dropped. When it is ``False`` every changed path
+is present with an exact ``occurrence_count``. When it is ``True`` the walk stopped early: some
+paths are absent rather than unchanged, and every ``occurrence_count`` is a lower bound, since
+occurrences after the stop were never reached.
+
+Typed dictionaries for the result and the change record are deliberately deferred -- the
+record shape varies with disclosure mode, so modelling it belongs with the REST layer that
+exposes it. :data:`DiffCategory` and :data:`DiffImpact` are stable enough to generate enums
+from today.
+"""
+
+from __future__ import annotations
+
+import copy
+import hashlib
+import json
+from collections.abc import Callable, Mapping, MutableMapping
+from contextlib import suppress
+from dataclasses import dataclass
+from datetime import timedelta
+from enum import Enum
+from typing import Any, Literal, NamedTuple
+
+import structlog
+
+from airflow.serialization.definitions.baseoperator import SerializedBaseOperator
+from airflow.serialization.definitions.mappedoperator import SerializedMappedOperator
+from airflow.serialization.serialized_objects import (
+ _DAG_CALLBACK_FIELDS,
+ _OPERATOR_TIMEDELTA_FIELDS,
+ DagSerialization,
+ OperatorSerialization,
+)
+
+log = structlog.get_logger(__name__)
+
+DIFF_SCHEMA_VERSION = 1
+DEFAULT_MAX_CHANGES = 500
+MAX_ALLOWED_CHANGES = 5000
+SUPPORTED_SERIALIZED_DAG_SCHEMA_VERSIONS = frozenset((1, 2, 3))
+
+DiffCategory = Literal[
+ "asset",
+ "authorization",
+ "callback",
+ "deadline",
+ "dependency",
+ "metadata",
+ "param",
+ "provenance",
+ "schedule",
+ "task",
+ "unknown",
+]
+DiffImpact = Literal["authorization", "execution", "metadata", "provenance", "unknown"]
+
+_ORDER_INSENSITIVE_LIST_PATHS = {
+ ("dag", "tags"),
+ ("dag", "allowed_run_types"),
+}
+_KEYED_COLLECTION_PATHS = {
+ ("dag", "tasks"),
+ ("dag", "dag_dependencies"),
+ *_ORDER_INSENSITIVE_LIST_PATHS,
+}
+# Canonicalization folds away __version and client_defaults and adds provenance, so these are the
+# only top-level sections the walk is allowed to name.
+_DIFF_V1_PUBLIC_ROOT_FIELDS = frozenset({"dag", "provenance"})
+_CUSTOM_TASK_FIELDS_PATH_COMPONENT = "custom_fields"
+# This allowlist is part of diff schema v1. Serializer schema changes must not
+# silently change the paths visible to callers of the diff API.
+_DIFF_V1_PUBLIC_TASK_FIELDS = frozenset(
+ {
+ "__type",
+ "_can_skip_downstream",
+ "_disallow_kwargs_override",
+ "_expand_input_attr",
+ "_is_empty",
+ "_is_mapped",
+ "_is_sensor",
+ "_logger_name",
+ "_needs_expansion",
+ "_operator_extra_links",
+ "_operator_name",
+ "_task_display_name",
+ "_task_module",
+ "allow_nested_operators",
+ "depends_on_past",
+ "do_xcom_push",
+ "doc",
+ "doc_json",
+ "doc_md",
+ "doc_rst",
+ "doc_yaml",
+ "downstream_task_ids",
+ "email",
+ "email_on_failure",
+ "email_on_retry",
+ "end_date",
+ "execution_timeout",
+ "executor",
+ "executor_config",
+ "expand_input",
+ "has_on_execute_callback",
+ "has_on_failure_callback",
+ "has_on_retry_callback",
+ "has_on_skipped_callback",
+ "has_on_success_callback",
+ "has_retry_policy",
+ "ignore_first_depends_on_past",
+ "inlets",
+ "is_setup",
+ "is_stub",
+ "is_teardown",
+ "map_index_template",
+ "max_active_tis_per_dag",
+ "max_active_tis_per_dagrun",
+ "max_retry_delay",
+ "multiple_outputs",
+ "on_failure_fail_dagrun",
+ "op_kwargs_expand_input",
+ "outlets",
+ "owner",
+ "params",
+ "partial_kwargs",
+ "pool",
+ "pool_slots",
+ "priority_weight",
+ "python_callable_name",
+ "queue",
+ "render_template_as_native_obj",
+ "reschedule",
+ "resources",
+ "retries",
+ "retry_delay",
+ "retry_exponential_backoff",
+ "run_as_user",
+ "start_date",
+ "start_from_trigger",
+ "start_trigger_args",
+ "task_id",
+ "task_type",
+ "template_ext",
+ "template_fields",
+ "template_fields_renderers",
+ "trigger_rule",
+ "ui_color",
+ "ui_fgcolor",
+ "wait_for_downstream",
+ "wait_for_past_depends_before_skipping",
+ "weight_rule",
+ }
+)
+_DIFF_V1_REDACTED_SCHEMA_TASK_FIELDS = frozenset({"_arg_bindings"})
+# Task-field categories used by _get_category.
+_DIFF_V1_TASK_ASSET_FIELDS = frozenset({"inlets", "outlets"})
+_DIFF_V1_TASK_PARAM_FIELDS = frozenset({"params"})
+_DIFF_V1_TASK_DEPENDENCY_FIELDS = frozenset({"downstream_task_ids"})
+_DIFF_V1_TASK_METADATA_FIELDS = frozenset(
+ {
+ "_operator_name",
+ "doc",
+ "doc_json",
+ "doc_md",
+ "doc_rst",
+ "doc_yaml",
+ "owner",
+ "ui_color",
+ "ui_fgcolor",
+ "_task_display_name",
+ "task_display_name",
+ }
+)
+_DIFF_V1_PUBLIC_PARTIAL_TASK_FIELDS = _DIFF_V1_PUBLIC_TASK_FIELDS | {"task_display_name"}
+_DEFAULT_ARGS_PATH = ("dag", "default_args")
+_RETRY_BACKOFF_FIELD = "retry_exponential_backoff"
+# Dag fields the schema gives no default for, added after the first serializer versions.
+_ABSENT_AS_NULL_DAG_FIELDS = ("allowed_run_types", "deadline")
+# These hydrate to sets (_deserialize_operator_field for a task, TaskGroup.*_ids.update for a group),
+# so a reordered stored list means the same edges and must not read as a dependency change.
+_SET_VALUED_ID_FIELDS = frozenset(
+ {"downstream_task_ids", "upstream_task_ids", "downstream_group_ids", "upstream_group_ids"}
+)
+# A Dag's default_args holds the same operator __init__ kwargs partial_kwargs holds -- the
+# serializer rewrites _HAS_FLAG_FIELDS in both -- so it reuses that allowlist. "__type" names the
+# stored value's own dict envelope rather than a key inside it.
+_DIFF_V1_PUBLIC_DEFAULT_ARGS_FIELDS = _DIFF_V1_PUBLIC_PARTIAL_TASK_FIELDS - {"__type"}
+# Classify every Dag schema field explicitly so new fields require a policy decision.
+_DIFF_V1_DAG_FIELD_CATEGORIES: dict[str, DiffCategory] = {
+ "_concurrency": "schedule",
+ "_processor_dags_folder": "provenance",
+ "access_control": "authorization",
+ "allowed_run_types": "schedule",
+ "bundle_name": "provenance",
+ "catchup": "schedule",
+ "dag_dependencies": "dependency",
+ "dag_display_name": "metadata",
+ "dag_id": "metadata",
+ "dagrun_timeout": "schedule",
+ "deadline": "deadline",
+ # Only reached when the stored value is not the dict envelope _collect_default_args_changes
+ # walks per key; an unreadable envelope keeps the conservative impact of its widest key.
+ "default_args": "task",
+ "description": "metadata",
+ # Decides whether a run pins a bundle version, so it changes which code later runs execute
+ # and whether triggering a historical version is allowed at all.
+ "disable_bundle_versioning": "task",
+ "doc_md": "metadata",
+ "edge_info": "metadata",
+ "end_date": "schedule",
+ "fail_fast": "schedule",
+ "fileloc": "provenance",
+ "has_on_failure_callback": "callback",
+ "has_on_success_callback": "callback",
+ "is_paused_upon_creation": "schedule",
+ "max_active_runs": "schedule",
+ "max_active_tasks": "schedule",
+ "max_consecutive_failed_dag_runs": "schedule",
+ "owner_links": "metadata",
+ "params": "param",
+ "relative_fileloc": "provenance",
+ "render_template_as_native_obj": "task",
+ "rerun_with_latest_version": "task",
+ "start_date": "schedule",
+ "tags": "metadata",
+ "task_group": "task",
+ "tasks": "task",
+ "timetable": "schedule",
+ "timezone": "schedule",
+}
+# Fields dropped from the current schema that a stored payload can still carry. Every entry must
+# survive _canonicalize_payload_v1, or it classifies nothing.
+_DIFF_V1_LEGACY_DAG_FIELD_CATEGORIES: dict[str, DiffCategory] = {
+ "fail_stop": "schedule",
+ "on_failure_callback": "callback",
+ "on_success_callback": "callback",
+ "schedule": "schedule",
+}
+_RECURSIVE_MAPPING_PATHS = {
+ (),
+ ("dag",),
+ ("provenance",),
+ *_KEYED_COLLECTION_PATHS,
+}
+_DIFF_V1_TASK_GROUP_METADATA_FIELDS = frozenset(
+ {"group_display_name", "tooltip", "doc_md", "ui_color", "ui_fgcolor"}
+)
+_DIFF_V1_PUBLIC_TASK_GROUP_FIELDS = _DIFF_V1_TASK_GROUP_METADATA_FIELDS | {
+ "_group_id",
+ "prefix_group_id",
+ "children",
+ "upstream_group_ids",
+ "downstream_group_ids",
+ "upstream_task_ids",
+ "downstream_task_ids",
+ "expand_input",
+ "is_mapped",
+}
+
+
+def build_unavailable_dag_diff(
+ *,
+ base_data: dict[str, Any] | None,
+ target_data: dict[str, Any] | None,
+ reason: str,
+) -> dict[str, Any]:
+ """Report a known unavailable reason without comparing the stored payloads."""
+ return _mark_unavailable(
+ _build_diff_result(_get_schema_version(base_data), _get_schema_version(target_data)), reason
+ )
+
+
+def build_serialized_dag_diff(
+ *,
+ base_data: dict[str, Any] | None,
+ target_data: dict[str, Any] | None,
+ base_provenance: Mapping[str, Any] | None = None,
+ target_provenance: Mapping[str, Any] | None = None,
+ include_values: bool = False,
+ max_changes: int = DEFAULT_MAX_CHANGES,
+) -> dict[str, Any]:
+ """
+ Compare two serialized Dag payloads and their provenance deterministically.
+
+ Callers must authorize disclosure of the entire serialized payload, including
+ access-control roles and permissions, before setting ``include_values=True``.
+ This exposes canonicalized values, digests, and identifying path components.
+
+ ``max_changes`` limits underlying changes admitted to the result. Redacted changes
+ with the same public path and operation share a record. Repeats can be counted past
+ the limit until a change needing a new record stops the walk. When ``truncated`` is
+ true, some paths are absent and occurrence counts are lower bounds.
+
+ An ``unavailable`` result includes the reason comparison failed.
+ """
+ validate_max_changes(max_changes)
+
+ base_schema_version = _get_schema_version(base_data)
+ target_schema_version = _get_schema_version(target_data)
+ result = _build_diff_result(base_schema_version, target_schema_version)
+
+ if base_data is None or target_data is None:
+ return _mark_unavailable(result, "serialized_dag_missing")
+
+ if base_schema_version is None or target_schema_version is None:
+ return _mark_unavailable(result, "serialized_dag_schema_version_missing")
+
+ unsupported_versions = [
+ version
+ for version in (base_schema_version, target_schema_version)
+ if version not in SUPPORTED_SERIALIZED_DAG_SCHEMA_VERSIONS
+ ]
+ if unsupported_versions:
+ return _mark_unavailable(
+ result, f"unsupported_serialized_dag_schema_version:{unsupported_versions[0]}"
+ )
+
+ try:
+ base_document = _canonicalize_payload_v1(base_data)
+ target_document = _canonicalize_payload_v1(target_data)
+ base_document["provenance"] = _canonicalize_value(dict(base_provenance or {}), path=("provenance",))
+ target_document["provenance"] = _canonicalize_value(
+ dict(target_provenance or {}), path=("provenance",)
+ )
+ except (AttributeError, KeyError, OverflowError, RecursionError, TypeError, ValueError) as error:
+ log.warning(
+ "Serialized Dag diff canonicalization failed",
+ error_type=type(error).__name__,
+ base_schema_version=base_schema_version,
+ target_schema_version=target_schema_version,
+ )
+ return _mark_unavailable(result, "serialized_dag_canonicalization_failed")
+
+ collector = _ChangeCollector(max_changes=max_changes, include_values=include_values)
+ try:
+ _collect_changes(base_document, target_document, path=(), collector=collector)
+ except RecursionError:
+ log.warning(
+ "Serialized Dag diff recursion limit exceeded",
+ base_schema_version=base_schema_version,
+ target_schema_version=target_schema_version,
+ )
+ return _mark_unavailable(result, "serialized_dag_recursion_limit_exceeded")
Review Comment:
I do not think this reason can reach a caller, and the mechanism I gave for it last round was wrong. Canonicalization runs first over a strict superset of what the walk recurses into, so it exhausts the stack first in every shape I could build. Swept at this HEAD in breeze at the default limit of 1000: a deeply nested `executor_config` returns `observed_state` through depth 488 and `serialized_dag_canonicalization_failed` from 490 on, and nested task groups, the one place the walk itself recurses without bound, are clean through depth 164 and `serialized_dag_canonicalization_failed` from 165 on. There is no band where the walk fails first. Both recursion tests mock the function they exercise (`_collect_changes` on line 1896, `_canonicalize_payload_v1` on line 1873), so neither observed which reason a real deep payload produces.
Deep payloads still degrade correctly, so this is about the wire contract rather than behaviour: line 53 lists `serialized_dag_recursion_limit_exceeded` among the reasons a client can receive, and the CLI and endpoint PRs will enumerate it from there. Either drop the handler with the bullet, or keep it and say in the docstring that it is defence in depth rather than a reachable reason.
##########
airflow-core/src/airflow/serialization/dag_version_diff.py:
##########
@@ -0,0 +1,1281 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""
+Observed-state diffs for serialized Dag payloads.
+
+Diff schema v1 -- the wire format every entry point here returns.
+:func:`build_serialized_dag_diff` and :func:`build_unavailable_dag_diff` both return one
+dictionary in the shape described below, and ``DIFF_SCHEMA_VERSION`` is the version of that
+shape: bump it whenever a client-observable part of this contract changes.
+
+Top-level keys, all always present:
+
+* ``diff_schema_version`` -- ``int``, the value of ``DIFF_SCHEMA_VERSION``.
+* ``serialized_dag_schema_versions`` -- ``{"base": int | None, "target": int | None}``, the
+ ``__version`` each stored payload carried. ``None`` means the key was absent or not an
+ integer; ``bool`` is rejected, so ``__version: true`` reads as ``None`` and never as
+ version 1. A version in ``SUPPORTED_SERIALIZED_DAG_SCHEMA_VERSIONS`` is upgraded to the
+ newest supported version before anything is compared.
+* ``mode`` -- ``"observed_state"`` when the two payloads were compared, ``"unavailable"``
+ when no comparison happened.
+* ``changes`` -- the change records described below, ordered by the deterministic walk.
+ Always empty when ``mode`` is ``"unavailable"``.
+* ``truncated`` -- ``True`` when more underlying changes exist than ``max_changes`` allowed.
+ Always ``False`` when ``mode`` is ``"unavailable"``.
+* ``values`` -- ``{"status": "available" | "unavailable"}``. ``"available"`` only for an
+ ``"observed_state"`` result whose caller authorized value disclosure, so
+ ``result["values"]["status"]`` is safe to read on every result of every entry point.
+
+``unavailable_reason`` is the one conditional key: present exactly when ``mode`` is
+``"unavailable"``. :func:`build_serialized_dag_diff` produces:
+
+* ``serialized_dag_missing`` -- one side has no stored payload.
+* ``serialized_dag_schema_version_missing`` -- one side carries no usable ``__version``.
+* ``unsupported_serialized_dag_schema_version:<n>`` -- ``<n>`` is the first unsupported
+ version found, base before target.
+* ``serialized_dag_canonicalization_failed`` -- a payload could not be normalized (malformed
+ structure, an unsupported ``client_defaults`` section, an unencodable value).
+* ``serialized_dag_recursion_limit_exceeded`` -- the comparison walk ran too deep.
+* ``serialized_dag_json_encoding_failed`` -- a value could not be encoded as canonical JSON.
+
+:meth:`~airflow.models.dag_version.DagVersion.get_diff` reuses
+``serialized_dag_decode_failed``, for a stored payload that will not decompress or parse, and
+``deadline_alert_missing``, for a payload that references a deadline alert row that is gone.
+:func:`build_unavailable_dag_diff` passes its caller's reason through unchanged, so a new
+caller extends this list rather than inventing an undocumented value.
+
+Each change record carries these keys in both disclosure modes:
+
+* ``path`` -- a JSON-Pointer-style path into a synthetic document with two roots: ``/dag``
+ (the canonicalized serialized Dag, with ``__version`` dropped and client defaults folded
+ into the tasks) and ``/provenance`` (the provenance mapping the caller supplied --
+ ``bundle_name``, ``bundle_version`` and ``version_data`` from ``get_diff``). ``~`` and
+ ``/`` inside a component are escaped as ``~0`` and ``~1``.
+* ``operation`` -- ``"added"``, ``"removed"`` or ``"changed"``.
+* ``category`` -- a :data:`DiffCategory`: ``task`` for task definitions and the Dag fields
+ that shape task execution, ``dependency`` for ``dag_dependencies`` and downstream task
+ ids, ``schedule`` for timetable, dates and concurrency, ``param`` for params,
+ ``asset`` for task inlets and outlets, ``deadline`` for deadline alerts,
+ ``callback`` for callback presence, ``metadata`` for descriptive and display fields,
+ ``authorization`` for ``access_control``, ``provenance`` for everything under
+ ``/provenance`` plus the file and bundle locators, and ``unknown`` for anything
+ unclassified. ``_DIFF_V1_DAG_FIELD_CATEGORIES`` is the authoritative Dag-field mapping;
+ an unclassified task field falls back to ``task``, which over-reports impact.
+ ``default_args`` is the one Dag field classified per key rather than as a whole, because its
+ keys are the operator fields they will be applied to: an allowlisted key takes that field's
+ category (``owner`` is ``metadata``, ``retries`` is ``task``), and the aggregate record for
+ the rest keeps the conservative ``task``. Only a stored value that is not a readable dict
+ envelope classifies as ``task`` as a whole.
+* ``impact`` -- a :data:`DiffImpact` derived from ``category``: ``provenance``, ``metadata``
+ and ``authorization`` carry through unchanged, every operational category becomes
+ ``execution``, and ``unknown`` stays ``unknown``.
+* ``occurrence_count`` -- how many underlying changes the record stands for.
+
+A caller that authorized values (``include_values=True``, ``values.status`` ``"available"``)
+additionally gets:
+
+* ``before_digest`` / ``after_digest`` -- ``"sha256:<hex>"`` over the canonical JSON of that
+ side, or ``None`` when that side is missing. Both keys are always present.
+* ``before_value`` / ``after_value`` -- the canonicalized value, which is the stored one after
+ schema and client defaults are folded in and params are normalized. ``before_value`` is absent from
+ an ``added`` record and ``after_value`` from a ``removed`` one, which is how a missing side
+ is told apart from a stored ``null``.
+
+``path`` and ``occurrence_count`` also differ between the modes. A redacted record reports the
+public path: members of keyed collections (``/dag/tasks``, ``/dag/dag_dependencies``,
+``/dag/tags``, ``/dag/allowed_run_types`` and task group children) collapse to ``*`` instead
+of naming a task, tag or group, and records sharing a public path and operation are merged
+into one whose ``occurrence_count`` counts the merged changes. An authorized record keeps the
+identifying component and always has an ``occurrence_count`` of 1. Fields outside the v1 task,
+task group and ``default_args`` allowlists are aggregated into a single ``custom_fields`` record
+in both modes, so neither a private serializer field nor a user-chosen ``default_args`` key is
+ever named by either -- ``/dag/default_args`` names only allowlisted operator field names, which
+is why its keys can be classified at all. A top-level key outside the v1 root allowlist is
+likewise reported as ``/custom_fields`` when redacted, while an authorized record keeps its real
+root path.
+
+``max_changes`` bounds the result rather than the traversal: both modes walk the same shape and
+reach the same changes, so authorization never decides which changes exist, and the public paths
+either mode reports -- and the order of their first occurrence -- are the same. What the bound
+admits does differ. An authorized record stands for one change, so the bound caps those directly.
+A redacted record stands for every change sharing its public path and operation, and a change
+reaching a record already in the result costs no new one, so redacted counting continues past the
+bound until a change needs a record the bound will not allow. The same payload at the same bound
+can therefore report more occurrences, and a different ``truncated``, when values are withheld.
+
+``truncated`` reports that such a change was dropped. When it is ``False`` every changed path
+is present with an exact ``occurrence_count``. When it is ``True`` the walk stopped early: some
+paths are absent rather than unchanged, and every ``occurrence_count`` is a lower bound, since
+occurrences after the stop were never reached.
+
+Typed dictionaries for the result and the change record are deliberately deferred -- the
+record shape varies with disclosure mode, so modelling it belongs with the REST layer that
+exposes it. :data:`DiffCategory` and :data:`DiffImpact` are stable enough to generate enums
+from today.
+"""
+
+from __future__ import annotations
+
+import copy
+import hashlib
+import json
+from collections.abc import Callable, Mapping, MutableMapping
+from contextlib import suppress
+from dataclasses import dataclass
+from datetime import timedelta
+from enum import Enum
+from typing import Any, Literal, NamedTuple
+
+import structlog
+
+from airflow.serialization.definitions.baseoperator import SerializedBaseOperator
+from airflow.serialization.definitions.mappedoperator import SerializedMappedOperator
+from airflow.serialization.serialized_objects import (
+ _DAG_CALLBACK_FIELDS,
+ _OPERATOR_TIMEDELTA_FIELDS,
+ DagSerialization,
+ OperatorSerialization,
+)
+
+log = structlog.get_logger(__name__)
+
+DIFF_SCHEMA_VERSION = 1
+DEFAULT_MAX_CHANGES = 500
+MAX_ALLOWED_CHANGES = 5000
+SUPPORTED_SERIALIZED_DAG_SCHEMA_VERSIONS = frozenset((1, 2, 3))
+
+DiffCategory = Literal[
+ "asset",
+ "authorization",
+ "callback",
+ "deadline",
+ "dependency",
+ "metadata",
+ "param",
+ "provenance",
+ "schedule",
+ "task",
+ "unknown",
+]
+DiffImpact = Literal["authorization", "execution", "metadata", "provenance", "unknown"]
+
+_ORDER_INSENSITIVE_LIST_PATHS = {
+ ("dag", "tags"),
+ ("dag", "allowed_run_types"),
+}
+_KEYED_COLLECTION_PATHS = {
+ ("dag", "tasks"),
+ ("dag", "dag_dependencies"),
+ *_ORDER_INSENSITIVE_LIST_PATHS,
+}
+# Canonicalization folds away __version and client_defaults and adds provenance, so these are the
+# only top-level sections the walk is allowed to name.
+_DIFF_V1_PUBLIC_ROOT_FIELDS = frozenset({"dag", "provenance"})
+_CUSTOM_TASK_FIELDS_PATH_COMPONENT = "custom_fields"
+# This allowlist is part of diff schema v1. Serializer schema changes must not
+# silently change the paths visible to callers of the diff API.
+_DIFF_V1_PUBLIC_TASK_FIELDS = frozenset(
+ {
+ "__type",
+ "_can_skip_downstream",
+ "_disallow_kwargs_override",
+ "_expand_input_attr",
+ "_is_empty",
+ "_is_mapped",
+ "_is_sensor",
+ "_logger_name",
+ "_needs_expansion",
+ "_operator_extra_links",
+ "_operator_name",
+ "_task_display_name",
+ "_task_module",
+ "allow_nested_operators",
+ "depends_on_past",
+ "do_xcom_push",
+ "doc",
+ "doc_json",
+ "doc_md",
+ "doc_rst",
+ "doc_yaml",
+ "downstream_task_ids",
+ "email",
+ "email_on_failure",
+ "email_on_retry",
+ "end_date",
+ "execution_timeout",
+ "executor",
+ "executor_config",
+ "expand_input",
+ "has_on_execute_callback",
+ "has_on_failure_callback",
+ "has_on_retry_callback",
+ "has_on_skipped_callback",
+ "has_on_success_callback",
+ "has_retry_policy",
+ "ignore_first_depends_on_past",
+ "inlets",
+ "is_setup",
+ "is_stub",
+ "is_teardown",
+ "map_index_template",
+ "max_active_tis_per_dag",
+ "max_active_tis_per_dagrun",
+ "max_retry_delay",
+ "multiple_outputs",
+ "on_failure_fail_dagrun",
+ "op_kwargs_expand_input",
+ "outlets",
+ "owner",
+ "params",
+ "partial_kwargs",
+ "pool",
+ "pool_slots",
+ "priority_weight",
+ "python_callable_name",
+ "queue",
+ "render_template_as_native_obj",
+ "reschedule",
+ "resources",
+ "retries",
+ "retry_delay",
+ "retry_exponential_backoff",
+ "run_as_user",
+ "start_date",
+ "start_from_trigger",
+ "start_trigger_args",
+ "task_id",
+ "task_type",
+ "template_ext",
+ "template_fields",
+ "template_fields_renderers",
+ "trigger_rule",
+ "ui_color",
+ "ui_fgcolor",
+ "wait_for_downstream",
+ "wait_for_past_depends_before_skipping",
+ "weight_rule",
+ }
+)
+_DIFF_V1_REDACTED_SCHEMA_TASK_FIELDS = frozenset({"_arg_bindings"})
+# Task-field categories used by _get_category.
+_DIFF_V1_TASK_ASSET_FIELDS = frozenset({"inlets", "outlets"})
+_DIFF_V1_TASK_PARAM_FIELDS = frozenset({"params"})
+_DIFF_V1_TASK_DEPENDENCY_FIELDS = frozenset({"downstream_task_ids"})
+_DIFF_V1_TASK_METADATA_FIELDS = frozenset(
+ {
+ "_operator_name",
+ "doc",
+ "doc_json",
+ "doc_md",
+ "doc_rst",
+ "doc_yaml",
+ "owner",
+ "ui_color",
+ "ui_fgcolor",
+ "_task_display_name",
+ "task_display_name",
+ }
+)
+_DIFF_V1_PUBLIC_PARTIAL_TASK_FIELDS = _DIFF_V1_PUBLIC_TASK_FIELDS | {"task_display_name"}
+_DEFAULT_ARGS_PATH = ("dag", "default_args")
+_RETRY_BACKOFF_FIELD = "retry_exponential_backoff"
+# Dag fields the schema gives no default for, added after the first serializer versions.
+_ABSENT_AS_NULL_DAG_FIELDS = ("allowed_run_types", "deadline")
+# These hydrate to sets (_deserialize_operator_field for a task, TaskGroup.*_ids.update for a group),
+# so a reordered stored list means the same edges and must not read as a dependency change.
+_SET_VALUED_ID_FIELDS = frozenset(
+ {"downstream_task_ids", "upstream_task_ids", "downstream_group_ids", "upstream_group_ids"}
+)
+# A Dag's default_args holds the same operator __init__ kwargs partial_kwargs holds -- the
+# serializer rewrites _HAS_FLAG_FIELDS in both -- so it reuses that allowlist. "__type" names the
+# stored value's own dict envelope rather than a key inside it.
+_DIFF_V1_PUBLIC_DEFAULT_ARGS_FIELDS = _DIFF_V1_PUBLIC_PARTIAL_TASK_FIELDS - {"__type"}
+# Classify every Dag schema field explicitly so new fields require a policy decision.
+_DIFF_V1_DAG_FIELD_CATEGORIES: dict[str, DiffCategory] = {
+ "_concurrency": "schedule",
+ "_processor_dags_folder": "provenance",
+ "access_control": "authorization",
+ "allowed_run_types": "schedule",
+ "bundle_name": "provenance",
+ "catchup": "schedule",
+ "dag_dependencies": "dependency",
+ "dag_display_name": "metadata",
+ "dag_id": "metadata",
+ "dagrun_timeout": "schedule",
+ "deadline": "deadline",
+ # Only reached when the stored value is not the dict envelope _collect_default_args_changes
+ # walks per key; an unreadable envelope keeps the conservative impact of its widest key.
+ "default_args": "task",
+ "description": "metadata",
+ # Decides whether a run pins a bundle version, so it changes which code later runs execute
+ # and whether triggering a historical version is allowed at all.
+ "disable_bundle_versioning": "task",
+ "doc_md": "metadata",
+ "edge_info": "metadata",
+ "end_date": "schedule",
+ "fail_fast": "schedule",
+ "fileloc": "provenance",
+ "has_on_failure_callback": "callback",
+ "has_on_success_callback": "callback",
+ "is_paused_upon_creation": "schedule",
+ "max_active_runs": "schedule",
+ "max_active_tasks": "schedule",
+ "max_consecutive_failed_dag_runs": "schedule",
+ "owner_links": "metadata",
+ "params": "param",
+ "relative_fileloc": "provenance",
+ "render_template_as_native_obj": "task",
+ "rerun_with_latest_version": "task",
+ "start_date": "schedule",
+ "tags": "metadata",
+ "task_group": "task",
+ "tasks": "task",
+ "timetable": "schedule",
+ "timezone": "schedule",
+}
+# Fields dropped from the current schema that a stored payload can still carry. Every entry must
+# survive _canonicalize_payload_v1, or it classifies nothing.
+_DIFF_V1_LEGACY_DAG_FIELD_CATEGORIES: dict[str, DiffCategory] = {
+ "fail_stop": "schedule",
+ "on_failure_callback": "callback",
+ "on_success_callback": "callback",
+ "schedule": "schedule",
+}
+_RECURSIVE_MAPPING_PATHS = {
+ (),
+ ("dag",),
+ ("provenance",),
+ *_KEYED_COLLECTION_PATHS,
+}
+_DIFF_V1_TASK_GROUP_METADATA_FIELDS = frozenset(
+ {"group_display_name", "tooltip", "doc_md", "ui_color", "ui_fgcolor"}
+)
+_DIFF_V1_PUBLIC_TASK_GROUP_FIELDS = _DIFF_V1_TASK_GROUP_METADATA_FIELDS | {
+ "_group_id",
+ "prefix_group_id",
+ "children",
+ "upstream_group_ids",
+ "downstream_group_ids",
+ "upstream_task_ids",
+ "downstream_task_ids",
+ "expand_input",
+ "is_mapped",
+}
+
+
+def build_unavailable_dag_diff(
+ *,
+ base_data: dict[str, Any] | None,
+ target_data: dict[str, Any] | None,
+ reason: str,
+) -> dict[str, Any]:
+ """Report a known unavailable reason without comparing the stored payloads."""
+ return _mark_unavailable(
+ _build_diff_result(_get_schema_version(base_data), _get_schema_version(target_data)), reason
+ )
+
+
+def build_serialized_dag_diff(
+ *,
+ base_data: dict[str, Any] | None,
+ target_data: dict[str, Any] | None,
+ base_provenance: Mapping[str, Any] | None = None,
+ target_provenance: Mapping[str, Any] | None = None,
+ include_values: bool = False,
+ max_changes: int = DEFAULT_MAX_CHANGES,
+) -> dict[str, Any]:
+ """
+ Compare two serialized Dag payloads and their provenance deterministically.
+
+ Callers must authorize disclosure of the entire serialized payload, including
+ access-control roles and permissions, before setting ``include_values=True``.
+ This exposes canonicalized values, digests, and identifying path components.
+
+ ``max_changes`` limits underlying changes admitted to the result. Redacted changes
+ with the same public path and operation share a record. Repeats can be counted past
+ the limit until a change needing a new record stops the walk. When ``truncated`` is
+ true, some paths are absent and occurrence counts are lower bounds.
+
+ An ``unavailable`` result includes the reason comparison failed.
+ """
+ validate_max_changes(max_changes)
+
+ base_schema_version = _get_schema_version(base_data)
+ target_schema_version = _get_schema_version(target_data)
+ result = _build_diff_result(base_schema_version, target_schema_version)
+
+ if base_data is None or target_data is None:
+ return _mark_unavailable(result, "serialized_dag_missing")
+
+ if base_schema_version is None or target_schema_version is None:
+ return _mark_unavailable(result, "serialized_dag_schema_version_missing")
+
+ unsupported_versions = [
+ version
+ for version in (base_schema_version, target_schema_version)
+ if version not in SUPPORTED_SERIALIZED_DAG_SCHEMA_VERSIONS
+ ]
+ if unsupported_versions:
+ return _mark_unavailable(
+ result, f"unsupported_serialized_dag_schema_version:{unsupported_versions[0]}"
+ )
+
+ try:
+ base_document = _canonicalize_payload_v1(base_data)
+ target_document = _canonicalize_payload_v1(target_data)
+ base_document["provenance"] = _canonicalize_value(dict(base_provenance or {}), path=("provenance",))
+ target_document["provenance"] = _canonicalize_value(
+ dict(target_provenance or {}), path=("provenance",)
+ )
+ except (AttributeError, KeyError, OverflowError, RecursionError, TypeError, ValueError) as error:
+ log.warning(
+ "Serialized Dag diff canonicalization failed",
+ error_type=type(error).__name__,
+ base_schema_version=base_schema_version,
+ target_schema_version=target_schema_version,
+ )
+ return _mark_unavailable(result, "serialized_dag_canonicalization_failed")
+
+ collector = _ChangeCollector(max_changes=max_changes, include_values=include_values)
+ try:
+ _collect_changes(base_document, target_document, path=(), collector=collector)
+ except RecursionError:
+ log.warning(
+ "Serialized Dag diff recursion limit exceeded",
+ base_schema_version=base_schema_version,
+ target_schema_version=target_schema_version,
+ )
+ return _mark_unavailable(result, "serialized_dag_recursion_limit_exceeded")
+ except _JsonEncodingError:
+ log.warning(
+ "Serialized Dag diff JSON encoding failed",
+ base_schema_version=base_schema_version,
+ target_schema_version=target_schema_version,
+ )
+ return _mark_unavailable(result, "serialized_dag_json_encoding_failed")
+
+ result["changes"] = collector.changes
+ result["truncated"] = collector.is_truncated
+ if include_values:
+ result["values"] = {"status": "available"}
+ return result
+
+
+class _ChangeCollector:
+ def __init__(self, *, max_changes: int, include_values: bool) -> None:
+ self.changes: list[dict[str, Any]] = []
+ self.count = 0
+ self.max_changes = max_changes
+ self.include_values = include_values
+ self._truncated = False
+ self._redacted_changes: dict[tuple[tuple[str, ...], str], dict[str, Any]] = {}
+
+ @property
+ def is_truncated(self) -> bool:
+ return self._truncated
+
+ def add(
+ self,
+ *,
+ path: tuple[str, ...],
+ operation: Literal["added", "removed", "changed"],
+ before: Any,
+ after: Any,
+ ) -> None:
+ self.count += 1
Review Comment:
`self.count += 1` sits above the merge early-return, so a repeat of an already-recorded public path spends budget without producing a record. That makes `max_changes` bound occurrences rather than records, and the redacted default then drops distinct paths.
Measured at this HEAD in breeze: 600 tasks all changing `retries` plus one `/dag/timezone` change, `max_changes=500`, redacted, returns a single record (`/dag/tasks/*/retries`, `occurrence_count` 600) with `truncated: true` and no timezone record at all. At 5000 tasks with `max_changes=5000`, which is `MAX_ALLOWED_CHANGES` and the most a caller may ask for, the answer is the same single record, so no permitted bound recovers the timezone change. The same 600-task case at `max_changes=5000` returns both paths with `truncated: false`, which is the shape line 112 promises ("bounds the result rather than the traversal").
I can see this is deliberate: lines 116-119 describe the counting continuing past the bound, and `test_build_diff_excludes_new_group_beyond_change_limit[False-2]` pins it. But it leaves the crowd-out that last round's grouping was asked to fix (#discussion_r4008627247, "the duplicates scale with task count rather than with content") in place, and harder to spot, because the result now reads as small rather than as repetitive. Moving `self.count += 1` below the repeat early-return makes the bound cap records. The redacted walk then stops truncating in practice, but its record count is bounded by the field allowlists rather than by task count, so that is a small set either way.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add serialized Dag version diff engine [airflow]
Posted by "kaxil (via GitHub)" <gi...@apache.org>.
kaxil commented on code in PR #69864:
URL: https://github.com/apache/airflow/pull/69864#discussion_r3962879415
##########
airflow-core/src/airflow/models/dag_version.py:
##########
@@ -245,6 +255,179 @@ def version(self) -> str:
"""A human-friendly representation of the version."""
return f"{self.dag_id}-{self.version_number}"
+ @classmethod
+ @provide_session
+ def get_diff(
+ cls,
+ dag_id: str,
+ base_version_number: int,
+ target_version_number: int,
+ *,
+ include_values: bool = False,
+ include_source: bool = False,
+ max_changes: int | None = None,
+ source_status: SourceStatus | None = None,
+ values_status: ValuesStatus | None = None,
+ session: Session = NEW_SESSION,
+ ) -> dict[str, Any]:
+ """
+ Compare two versions of a Dag using their currently stored state.
+
+ ``source_status`` is supplied by callers that have an authorization context. The CLI has
+ operator-level authority and leaves it unset, while API callers calculate it before this
+ method returns any source content. API callers use ``values_status`` to suppress raw values
+ when the user cannot access Dag code; the structural diff remains available in redacted form.
+ """
+ # Keep this local to avoid the dag_version -> dag_version_diff -> serialized_objects cycle.
+ from airflow.models.serialized_dag import SerializedDagModel
+ from airflow.serialization.dag_version_diff import (
+ DEFAULT_MAX_CHANGES,
+ build_serialized_dag_diff,
+ )
+
+ if source_status is not None and source_status not in _VALID_SOURCE_STATUSES:
+ raise ValueError(f"source_status must be one of {_VALID_SOURCE_STATUSES}, not {source_status!r}")
+ if values_status is not None and values_status not in _VALID_VALUES_STATUSES:
+ raise ValueError(f"values_status must be one of {_VALID_VALUES_STATUSES}, not {values_status!r}")
+
+ if max_changes is None:
+ max_changes = DEFAULT_MAX_CHANGES
+ if base_version_number < 1 or target_version_number < 1:
+ raise ValueError("Dag version numbers must be positive integers")
+
+ if source_status is None:
+ source_status = "current_stored_code" if include_source else "unavailable"
+
+ query = (
+ select(cls)
+ .where(
+ cls.dag_id == dag_id,
+ cls.version_number.in_((base_version_number, target_version_number)),
+ )
+ .options(joinedload(cls.serialized_dag).selectinload(SerializedDagModel.deadline_alerts))
+ )
+ if include_source and source_status == "current_stored_code":
+ query = query.options(joinedload(cls.dag_code))
+
+ versions = {version.version_number: version for version in session.scalars(query).all()}
+ missing_version = next(
+ (
+ version_number
+ for version_number in (base_version_number, target_version_number)
+ if version_number not in versions
+ ),
+ None,
+ )
+ if missing_version is not None:
+ raise DagVersionNotFound(
+ f"The DagVersion with dag_id: `{dag_id}` and version_number: `{missing_version}` was not found"
+ )
+
+ base_version = versions[base_version_number]
+ target_version = versions[target_version_number]
+ base_data, base_unavailable_reason = _get_serialized_diff_data(base_version.serialized_dag)
+ target_data, target_unavailable_reason = _get_serialized_diff_data(target_version.serialized_dag)
+ effective_include_values = include_values and values_status in {None, "available"}
+ result = build_serialized_dag_diff(
+ base_data=base_data,
+ target_data=target_data,
+ base_provenance=_get_provenance(base_version),
+ target_provenance=_get_provenance(target_version),
+ include_values=effective_include_values,
+ max_changes=max_changes,
+ )
+ if unavailable_reason := base_unavailable_reason or target_unavailable_reason:
+ result.update(
+ mode="unavailable", unavailable_reason=unavailable_reason, changes=[], truncated=False
+ )
+ if include_values:
+ values_available = effective_include_values and result["mode"] == "observed_state"
+ result["values"] = {"status": "available" if values_available else "unavailable"}
Review Comment:
`values` only shows up when `include_values` is set, while `source` is always present with `status: "unavailable"` when it's off. That means `result["source"]["status"]` is always safe to read but `result["values"]["status"]` raises KeyError on a redacted request. Emitting `values` unconditionally would keep the `diff_schema_version: 1` shape stable for clients.
##########
airflow-core/src/airflow/serialization/dag_version_diff.py:
##########
@@ -0,0 +1,655 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""Observed-state diffs for serialized Dag payloads."""
+
+from __future__ import annotations
+
+import copy
+import hashlib
+import json
+from collections.abc import Callable, Mapping
+from enum import Enum
+from typing import Any, Literal
+
+from airflow.serialization.definitions.baseoperator import SerializedBaseOperator
+from airflow.serialization.definitions.mappedoperator import SerializedMappedOperator
+from airflow.serialization.serialized_objects import (
+ _DAG_CALLBACK_FIELDS,
+ _OPERATOR_TIMEDELTA_FIELDS,
+ DagSerialization,
+ OperatorSerialization,
+)
+
+DIFF_SCHEMA_VERSION = 1
+DEFAULT_MAX_CHANGES = 500
+MAX_ALLOWED_CHANGES = 5000
+SUPPORTED_SERIALIZED_DAG_SCHEMA_VERSIONS = frozenset((1, 2, 3))
+
+DiffMode = Literal["observed_state", "unavailable"]
+
+_ORDER_INSENSITIVE_LIST_PATHS = {
+ ("dag", "tags"),
+ ("dag", "allowed_run_types"),
+}
+_KEYED_COLLECTION_PATHS = {
+ ("dag", "tasks"),
+ ("dag", "dag_dependencies"),
+ *_ORDER_INSENSITIVE_LIST_PATHS,
+}
+_CUSTOM_TASK_FIELDS_PATH_COMPONENT = "custom_fields"
+# This allowlist is part of diff schema v1. Serializer schema changes must not
+# silently change the paths visible to callers of the diff API.
+_DIFF_V1_PUBLIC_TASK_FIELDS = frozenset(
+ {
+ "__type",
+ "_disallow_kwargs_override",
+ "_expand_input_attr",
+ "_is_mapped",
+ "_is_sensor",
+ "_logger_name",
+ "_needs_expansion",
+ "_operator_extra_links",
+ "_task_display_name",
+ "_task_module",
+ "allow_nested_operators",
+ "depends_on_past",
+ "do_xcom_push",
+ "doc",
+ "doc_json",
+ "doc_md",
+ "doc_rst",
+ "doc_yaml",
+ "downstream_task_ids",
+ "email_on_failure",
+ "email_on_retry",
+ "end_date",
+ "execution_timeout",
+ "executor",
+ "executor_config",
+ "has_on_execute_callback",
+ "has_on_failure_callback",
+ "has_on_retry_callback",
+ "has_on_skipped_callback",
+ "has_on_success_callback",
+ "ignore_first_depends_on_past",
+ "inlets",
+ "is_setup",
+ "is_teardown",
+ "map_index_template",
+ "max_active_tis_per_dag",
+ "max_active_tis_per_dagrun",
+ "max_retry_delay",
+ "multiple_outputs",
+ "on_failure_fail_dagrun",
+ "outlets",
+ "owner",
+ "params",
+ "partial_kwargs",
+ "pool",
+ "pool_slots",
+ "priority_weight",
+ "queue",
+ "render_template_as_native_obj",
+ "retries",
+ "retry_delay",
+ "retry_exponential_backoff",
+ "start_date",
+ "start_from_trigger",
+ "start_trigger_args",
+ "task_id",
+ "task_type",
+ "template_ext",
+ "template_fields",
+ "template_fields_renderers",
+ "trigger_rule",
+ "ui_color",
+ "ui_fgcolor",
+ "wait_for_downstream",
+ "wait_for_past_depends_before_skipping",
+ "weight_rule",
+ }
+)
+_DIFF_V1_REDACTED_SCHEMA_TASK_FIELDS = frozenset({"_arg_bindings"})
+_DIFF_V1_PUBLIC_PARTIAL_TASK_FIELDS = _DIFF_V1_PUBLIC_TASK_FIELDS | {"task_display_name"}
+# Classify every Dag schema field explicitly so new fields require a policy decision.
+_DIFF_V1_DAG_FIELD_CATEGORIES = {
+ "_concurrency": "schedule",
+ "_processor_dags_folder": "provenance",
+ "access_control": "authorization",
+ "allowed_run_types": "schedule",
+ "bundle_name": "provenance",
+ "catchup": "schedule",
+ "dag_dependencies": "dependency",
+ "dag_display_name": "metadata",
+ "dag_id": "metadata",
+ "dagrun_timeout": "schedule",
+ "deadline": "deadline",
+ "default_args": "param",
+ "description": "metadata",
+ "disable_bundle_versioning": "task",
+ "doc_md": "metadata",
+ "edge_info": "metadata",
+ "end_date": "schedule",
+ "fail_fast": "schedule",
+ "fileloc": "provenance",
+ "has_on_failure_callback": "callback",
+ "has_on_success_callback": "callback",
+ "is_paused_upon_creation": "schedule",
+ "max_active_runs": "schedule",
+ "max_active_tasks": "schedule",
+ "max_consecutive_failed_dag_runs": "schedule",
+ "owner_links": "metadata",
+ "params": "param",
+ "relative_fileloc": "provenance",
+ "render_template_as_native_obj": "task",
+ "rerun_with_latest_version": "task",
+ "start_date": "schedule",
+ "tags": "metadata",
+ "task_group": "task",
+ "tasks": "task",
+ "timetable": "schedule",
+ "timezone": "schedule",
+}
+_DIFF_V1_LEGACY_DAG_FIELD_CATEGORIES = {
+ "fail_stop": "schedule",
+ "on_failure_callback": "callback",
+ "on_success_callback": "callback",
+ "schedule": "schedule",
+ "schedule_interval": "schedule",
+}
+_REDACTED_RECURSIVE_MAPPING_PATHS = {
+ (),
+ ("dag",),
+ ("dag", "task_group"),
+ ("provenance",),
+ *_KEYED_COLLECTION_PATHS,
+}
+
+
+def build_serialized_dag_diff(
+ *,
+ base_data: dict[str, Any] | None,
+ target_data: dict[str, Any] | None,
+ base_provenance: Mapping[str, Any] | None = None,
+ target_provenance: Mapping[str, Any] | None = None,
+ include_values: bool = False,
+ max_changes: int = DEFAULT_MAX_CHANGES,
+) -> dict[str, Any]:
+ """
+ Build a bounded, deterministic diff from two stored serialized Dag payloads.
+
+ Raw values, digests, and value-derived path components are returned only when
+ ``include_values`` is true.
+ """
+ _validate_max_changes(max_changes)
+
+ base_schema_version = _get_schema_version(base_data)
+ target_schema_version = _get_schema_version(target_data)
+ result: dict[str, Any] = {
+ "diff_schema_version": DIFF_SCHEMA_VERSION,
+ "serialized_dag_schema_versions": {
+ "base": base_schema_version,
+ "target": target_schema_version,
+ },
+ "mode": "observed_state",
+ "changes": [],
+ "truncated": False,
+ }
+
+ if base_data is None or target_data is None:
+ return _mark_unavailable(result, "serialized_dag_missing")
+
+ if base_schema_version is None or target_schema_version is None:
+ return _mark_unavailable(result, "serialized_dag_schema_version_missing")
+
+ unsupported_versions = [
+ version
+ for version in (base_schema_version, target_schema_version)
+ if version not in SUPPORTED_SERIALIZED_DAG_SCHEMA_VERSIONS
+ ]
+ if unsupported_versions:
+ return _mark_unavailable(
+ result, f"unsupported_serialized_dag_schema_version:{unsupported_versions[0]}"
+ )
+
+ try:
+ base_document = _canonicalize_payload_v1(base_data)
+ target_document = _canonicalize_payload_v1(target_data)
+ except (AttributeError, KeyError, OverflowError, TypeError, ValueError):
+ return _mark_unavailable(result, "serialized_dag_canonicalization_failed")
+
+ base_document["provenance"] = _canonicalize_value(dict(base_provenance or {}), path=("provenance",))
+ target_document["provenance"] = _canonicalize_value(dict(target_provenance or {}), path=("provenance",))
+
+ collector = _ChangeCollector(max_changes=max_changes, include_values=include_values)
+ _collect_changes(base_document, target_document, path=(), collector=collector)
+ result["changes"] = collector.changes
+ result["truncated"] = collector.is_truncated
+ return result
+
+
+class _ChangeCollector:
+ def __init__(self, *, max_changes: int, include_values: bool) -> None:
+ self.changes: list[dict[str, Any]] = []
+ self.count = 0
+ self.max_changes = max_changes
+ self.include_values = include_values
+
+ @property
+ def is_truncated(self) -> bool:
+ return self.count > self.max_changes
+
+ def add(
+ self,
+ *,
+ path: tuple[str, ...],
+ operation: Literal["added", "removed", "changed"],
+ before: Any,
+ after: Any,
+ ) -> None:
+ self.count += 1
+ if len(self.changes) >= self.max_changes:
+ return
+
+ public_path = _get_public_path(path)
+ category = _get_category(public_path)
+ change: dict[str, Any] = {
+ "path": _format_path(path if self.include_values else public_path),
+ "operation": operation,
+ "category": category,
+ "impact": _get_impact(category),
+ }
+ if self.include_values:
+ change["before_digest"] = None if before is _MISSING else _get_digest(before)
+ change["after_digest"] = None if after is _MISSING else _get_digest(after)
+ if before is not _MISSING:
+ change["before_value"] = before
+ if after is not _MISSING:
+ change["after_value"] = after
+ self.changes.append(change)
+
+
+_MISSING = object()
+
+
+def _validate_max_changes(max_changes: int) -> None:
+ if max_changes < 1:
+ raise ValueError("max_changes must be a positive integer")
+ if max_changes > MAX_ALLOWED_CHANGES:
+ raise ValueError(f"max_changes must not exceed {MAX_ALLOWED_CHANGES}")
+
+
+def _get_schema_version(data: Mapping[str, Any] | None) -> int | None:
+ if not isinstance(data, Mapping):
+ return None
+ version = data.get("__version")
+ return version if isinstance(version, int) and not isinstance(version, bool) else None
+
+
+def _mark_unavailable(result: dict[str, Any], reason: str) -> dict[str, Any]:
+ result["mode"] = "unavailable"
+ result["unavailable_reason"] = reason
+ return result
+
+
+def _canonicalize_payload_v1(data: dict[str, Any]) -> dict[str, Any]:
+ payload = copy.deepcopy(data)
+ version = _get_schema_version(payload)
+ if version is None:
+ raise ValueError("missing or invalid __version")
+ if version == 1:
+ DagSerialization.conversion_v1_to_v2(payload)
+ DagSerialization.conversion_v2_to_v3(payload)
+ elif version == 2:
+ DagSerialization.conversion_v2_to_v3(payload)
+ if not isinstance(payload.get("dag"), Mapping):
+ raise ValueError("missing dag object")
+ dag_defaults = {
+ field: value
+ for field, value in DagSerialization.get_schema_defaults("dag").items()
+ # Dag callback flags are enabled by their presence, even when their value is false.
+ if field not in _DAG_CALLBACK_FIELDS
+ }
+ payload["dag"] = {**dag_defaults, **payload["dag"]}
+ _apply_task_defaults(payload)
+ payload.pop("__version", None)
+ return _canonicalize_value(payload, path=())
+
+
+def _apply_task_defaults(payload: dict[str, Any]) -> None:
+ client_defaults = payload.pop("client_defaults", None)
+ if client_defaults is None:
+ client_defaults = {}
+ if not isinstance(client_defaults, Mapping):
+ raise ValueError("client_defaults is not an object")
+
+ task_defaults = client_defaults.get("tasks", {})
+ if not isinstance(task_defaults, Mapping):
+ raise ValueError("client_defaults.tasks is not an object")
+
+ schema_defaults = DagSerialization.get_schema_defaults("operator")
+ partial_fields = (
+ SerializedBaseOperator.get_serialized_fields() - SerializedMappedOperator.get_serialized_fields()
+ )
+ tasks = payload["dag"].get("tasks", [])
+ if not isinstance(tasks, list):
+ raise ValueError("dag.tasks is not a list")
+ for task in tasks:
+ if not isinstance(task, dict) or not isinstance(task.get("__var"), Mapping):
+ raise ValueError("task entry is not an object")
+ encoded_task = OperatorSerialization._apply_defaults_to_encoded_op(
+ dict(task["__var"]), dict(client_defaults)
+ )
+ task_data = {**schema_defaults, **OperatorSerialization._upgrade_encoded_operator(encoded_task)}
+ if task_data.get("_is_mapped"):
+ partial_kwargs = task_data.get("partial_kwargs", {})
+ if not isinstance(partial_kwargs, Mapping):
+ raise ValueError("partial_kwargs is not an object")
+ effective_partial_kwargs = dict(task_defaults)
Review Comment:
I think the precedence is inverted here for mapped tasks. `task_data` already has `client_defaults.tasks` merged underneath it by `_apply_defaults_to_encoded_op` above, so seeding from `task_defaults` puts the client default in first, and the `setdefault` on line 367 can then never replace it with the explicit top-level value. `populate_operator` resolves the other way: `partial_kwargs` wins, then the top-level value, then the client default.
The effect is that real changes go missing rather than showing up wrong. With `client_defaults.tasks = {"retries": 5}` and two mapped-task payloads differing only in top-level `retries: 2` vs `retries: 3`, `build_serialized_dag_diff` returns an empty `changes` list, while `populate_operator` resolves 2 and 3. `pool` behaves the same way.
`effective_partial_kwargs = dict(partial_kwargs)` (dropping the `task_defaults` seed and the `update`) matched `populate_operator` on every case I tried, and the whole `test_dag_version_diff.py` suite still passes with it, which suggests a gap: the mapped-default tests only shadow through `partial_kwargs`, never through a top-level field.
##########
airflow-core/src/airflow/models/dag_version.py:
##########
@@ -245,6 +255,179 @@ def version(self) -> str:
"""A human-friendly representation of the version."""
return f"{self.dag_id}-{self.version_number}"
+ @classmethod
+ @provide_session
+ def get_diff(
+ cls,
+ dag_id: str,
+ base_version_number: int,
+ target_version_number: int,
+ *,
+ include_values: bool = False,
+ include_source: bool = False,
+ max_changes: int | None = None,
+ source_status: SourceStatus | None = None,
+ values_status: ValuesStatus | None = None,
+ session: Session = NEW_SESSION,
+ ) -> dict[str, Any]:
+ """
+ Compare two versions of a Dag using their currently stored state.
+
+ ``source_status`` is supplied by callers that have an authorization context. The CLI has
+ operator-level authority and leaves it unset, while API callers calculate it before this
+ method returns any source content. API callers use ``values_status`` to suppress raw values
+ when the user cannot access Dag code; the structural diff remains available in redacted form.
+ """
+ # Keep this local to avoid the dag_version -> dag_version_diff -> serialized_objects cycle.
+ from airflow.models.serialized_dag import SerializedDagModel
+ from airflow.serialization.dag_version_diff import (
+ DEFAULT_MAX_CHANGES,
+ build_serialized_dag_diff,
+ )
+
+ if source_status is not None and source_status not in _VALID_SOURCE_STATUSES:
+ raise ValueError(f"source_status must be one of {_VALID_SOURCE_STATUSES}, not {source_status!r}")
+ if values_status is not None and values_status not in _VALID_VALUES_STATUSES:
+ raise ValueError(f"values_status must be one of {_VALID_VALUES_STATUSES}, not {values_status!r}")
+
+ if max_changes is None:
+ max_changes = DEFAULT_MAX_CHANGES
Review Comment:
`max_changes` isn't range-checked until `build_serialized_dag_diff` runs, so an out-of-range value pays for the version lookup and both serialized payload loads before raising. Calling `_validate_max_changes` here instead would reject it before any DB work, which will matter more once `max_changes` comes in as a query param.
##########
airflow-core/src/airflow/models/dag_version.py:
##########
@@ -245,6 +255,179 @@ def version(self) -> str:
"""A human-friendly representation of the version."""
return f"{self.dag_id}-{self.version_number}"
+ @classmethod
+ @provide_session
+ def get_diff(
+ cls,
+ dag_id: str,
+ base_version_number: int,
+ target_version_number: int,
+ *,
+ include_values: bool = False,
+ include_source: bool = False,
+ max_changes: int | None = None,
+ source_status: SourceStatus | None = None,
+ values_status: ValuesStatus | None = None,
+ session: Session = NEW_SESSION,
+ ) -> dict[str, Any]:
+ """
+ Compare two versions of a Dag using their currently stored state.
+
+ ``source_status`` is supplied by callers that have an authorization context. The CLI has
+ operator-level authority and leaves it unset, while API callers calculate it before this
+ method returns any source content. API callers use ``values_status`` to suppress raw values
+ when the user cannot access Dag code; the structural diff remains available in redacted form.
+ """
+ # Keep this local to avoid the dag_version -> dag_version_diff -> serialized_objects cycle.
+ from airflow.models.serialized_dag import SerializedDagModel
+ from airflow.serialization.dag_version_diff import (
+ DEFAULT_MAX_CHANGES,
+ build_serialized_dag_diff,
+ )
+
+ if source_status is not None and source_status not in _VALID_SOURCE_STATUSES:
+ raise ValueError(f"source_status must be one of {_VALID_SOURCE_STATUSES}, not {source_status!r}")
+ if values_status is not None and values_status not in _VALID_VALUES_STATUSES:
+ raise ValueError(f"values_status must be one of {_VALID_VALUES_STATUSES}, not {values_status!r}")
+
+ if max_changes is None:
+ max_changes = DEFAULT_MAX_CHANGES
+ if base_version_number < 1 or target_version_number < 1:
+ raise ValueError("Dag version numbers must be positive integers")
+
+ if source_status is None:
+ source_status = "current_stored_code" if include_source else "unavailable"
Review Comment:
This defaults to full disclosure when a caller doesn't pass `source_status`, and `values_status=None` on line 330 does the same for raw values. An API handler that wires `include_source` to a query param but forgets to compute `source_status` would serve DAG source to whoever can reach the endpoint, and nothing in here would catch it.
Given the REST endpoint is the next PR in the stack, is it worth making `source_status` required and having the CLI pass `"current_stored_code"` explicitly? Omitting it becomes a TypeError rather than a disclosure. This is separate from the validation point in #discussion_r3580125653, which was about mistyped status values rather than the default direction.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add serialized Dag version diff engine [airflow]
Posted by "ephraimbuddy (via GitHub)" <gi...@apache.org>.
ephraimbuddy merged PR #69864:
URL: https://github.com/apache/airflow/pull/69864
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org