You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@superset.apache.org by "mikebridge (via GitHub)" <gi...@apache.org> on 2026/06/15 21:16:11 UTC
[PR] feat(versioning): cross-entity version activity view [superset]
mikebridge opened a new pull request, #41076:
URL: https://github.com/apache/superset/pull/41076
### SUMMARY
Stacked follow-up to #39603 (entity versioning, [sc-103156](https://app.shortcut.com/preset/story/103156)). Adds a **cross-entity version activity view**. ([sc-107283](https://app.shortcut.com/preset/story/107283))
Given any dashboard, chart, or dataset, `GET /api/v1/{chart,dashboard,dataset}/<uuid>/activity/` returns a unified, time-ordered stream of version-change activity for the entity **and** the entities transitively related to it through the versioned dependency chain (dashboard → its charts → their datasets), time-windowed by when each relationship was active. It answers "what changed in this dashboard?" honestly — including a chart whose `params` changed or a dataset whose column was renamed underneath it.
Reads sc-103156's existing data foundation (`dashboard_slices_version`, `slices_version.datasource_id`, `version_changes`, `version_transaction`) — **no new tables/columns; additive `CREATE INDEX` only, if any.**
> ⚠️ **Stacked PR.** Built on top of #39603. Until that merges, the diff below includes #39603's commits — review the commits / `superset/versioning/activity/` package specific to this branch. Contains one `debug(*)` throwaway (React shell + Flask route, `b4d6721993`) that reverts before merge.
**Merge order:** #39603 → **this PR** (either order relative to the retention PR, [sc-111099](https://app.shortcut.com/preset/story/111099)).
**What this adds:**
- `superset/versioning/activity/` package — core module, response schemas, and a query layer that joins through the M:N and datasource shadows + `version_changes`.
- Three `/activity/` endpoints (chart / dashboard / dataset), authorised via the resource's read permission with **SQL-side visibility filtering** so related-entity records the user can't see are dropped silently (standard RBAC).
- Three-dimension record consumption feeding version-history dropdown UI feedback (headlines, first-save marker, search), plus capture-path error metrics and phase-level observability signals.
**Deferred / out of scope:** `actor_kind` AI/bot badge (AV-014, no live consumer yet); external dataset events such as schema/data refresh (AV-017).
### BEFORE/AFTER SCREENSHOTS OR ANIMATED GIF
Backend + a throwaway demo shell only; final UI lands separately. (Mockups referenced in `specs/sc-107283-versioning-activity-view/`.)
### TESTING INSTRUCTIONS
```bash
# Generate history on a dashboard and its charts, then:
curl "http://localhost:8088/api/v1/dashboard/$DASHBOARD_UUID/activity/" | jq
# Related-entity entries (charts/datasets) appear inline, time-windowed by relationship validity.
pytest tests/integration_tests/versioning/ -v
```
### ADDITIONAL INFORMATION
- [x] Has associated issue: [sc-107283](https://app.shortcut.com/preset/story/107283)
- [ ] Required feature flags:
- [x] Changes UI (version-history dropdown consumption; throwaway demo shell reverts before merge)
- [ ] Includes DB Migration (follow approval process in [SIP-59](https://github.com/apache/superset/issues/13351))
- [x] Introduces new feature or API (3 new endpoints)
- [ ] Removes existing feature or API
Depends on #39603.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(versioning): cross-entity version activity view [superset]
Posted by "codeant-ai-for-open-source[bot] (via GitHub)" <gi...@apache.org>.
codeant-ai-for-open-source[bot] commented on code in PR #41076:
URL: https://github.com/apache/superset/pull/41076#discussion_r3560240437
##########
superset/versioning/activity/orchestrator.py:
##########
@@ -0,0 +1,434 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Top-level orchestrator + query-param parsing + observability.
+
+This is the public entry point for the activity-view read path. One
+function — :func:`get_activity` — dispatches on the path entity's
+model class to assemble the cross-entity activity stream:
+
+1. ``resolve_path_entity`` (queries.py) — resolve UUID → live entity.
+2. ``resolve_scope`` (scope.py) — build the related-entity window list.
+3. ``fetch_change_records`` (queries.py) — pull rows from
+ ``version_changes`` joined with ``version_transaction`` and ``ab_user``.
+4. ``filter_records_by_visibility`` (visibility.py) — silent
+ drop of records the requester can't read.
+5. ``apply_entity_name_denormalization`` (queries.py) — resolve entity names
+ from the shadow row valid at each record's transaction_id.
+6. ``apply_record_decoration`` (render.py) — synthesize the ActivityRecord
+ DTO fields and strip internal-only columns.
+7. Paginate in Python over the post-filter list.
+
+Parameter parsing for the REST endpoints lives here too —
+:func:`parse_activity_query_params` is called by the three
+``/activity/`` endpoint handlers before they call ``get_activity``.
+Same for the observability instrumentation: ``_phase_timer`` and
+``_emit_request_shape_attributes`` emit the per-phase timing and
+request-shape metrics, on the same prefix the cross-coupling test pins.
+"""
+
+from __future__ import annotations
+
+import contextlib
+from collections.abc import Iterator
+from datetime import datetime, timezone
+from typing import Any
+from uuid import UUID
+
+from flask import Response
+from flask_appbuilder import Model
+
+from superset.utils import json
+from superset.versioning.activity.kinds import EntityWindows
+from superset.versioning.activity.queries import (
+ apply_entity_name_denormalization,
+ fetch_change_records,
+ first_tracked_tx,
+ mark_first_tracked_saves,
+ resolve_path_entity,
+)
+from superset.versioning.activity.render import apply_record_decoration
+from superset.versioning.activity.scope import resolve_scope
+from superset.versioning.activity.visibility import filter_records_by_visibility
+from superset.versioning.api_helpers import (
+ PathEntityResponseError,
+ resolve_endpoint_path_entity,
+)
+
+_DEFAULT_PAGE_SIZE = 25
+_MAX_PAGE_SIZE = 200
+_VALID_INCLUDE_VALUES: frozenset[str] = frozenset({"self", "related", "all"})
+
+
+# Upper bound on the ``q`` search string. The search is a substring scan over
+# the (already-capped) materialized record set, so this is a cheap-DoS guard,
+# not a correctness limit.
+_MAX_Q_LENGTH = 1024
+
+
+class ActivityParamsError(ValueError):
+ """Raised by :func:`parse_activity_query_params` when a query param is
+ malformed. The endpoint catches this and maps to ``response_400``;
+ no other callers should depend on the exception type."""
+
+
+def parse_activity_query_params(args: Any) -> dict[str, Any]:
+ """Parse the ``since`` / ``until`` / ``include`` / ``page`` / ``page_size``
+ query parameters into the kwargs ``get_activity`` accepts.
+
+ Raises :class:`ActivityParamsError` (subclass of ``ValueError``) when
+ a parameter is malformed. Shared across the three endpoint families
+ (dashboards, charts, datasets) so the parsing and 400-messaging stay
+ consistent.
+ """
+ params: dict[str, Any] = {
+ "include": _parse_include(args.get("include", "all")),
+ "page": _parse_page(args.get("page", "0")),
+ "page_size": _parse_page_size(args.get("page_size")),
+ }
+ if (since := _parse_optional_iso(args.get("since"), name="since")) is not None:
+ params["since"] = since
+ if (until := _parse_optional_iso(args.get("until"), name="until")) is not None:
+ params["until"] = until
+ if q := (args.get("q") or "").strip():
+ if len(q) > _MAX_Q_LENGTH:
+ raise ActivityParamsError(f"'q' must be at most {_MAX_Q_LENGTH} characters")
+ params["q"] = q
+ return params
+
+
+def _parse_optional_iso(raw: str | None, *, name: str) -> datetime | None:
+ """Parse a missing-or-ISO-datetime field; ``None`` for missing,
+ ``ActivityParamsError`` for malformed."""
+ if not raw:
+ return None
+ parsed = _parse_iso_datetime(raw)
+ if parsed is None:
+ raise ActivityParamsError(f"Invalid {name!r} datetime: {raw!r}")
+ return parsed
+
+
+def _parse_include(value: str) -> str:
+ if value not in _VALID_INCLUDE_VALUES:
+ raise ActivityParamsError(
+ f"Invalid 'include' value: {value!r}; "
+ f"must be one of {sorted(_VALID_INCLUDE_VALUES)}"
+ )
+ return value
+
+
+def _parse_page(raw: str) -> int:
+ try:
+ value = int(raw)
+ except (TypeError, ValueError) as exc:
+ raise ActivityParamsError(f"Invalid 'page' value: {raw!r}") from exc
+ if value < 0:
+ raise ActivityParamsError("Invalid 'page' value: must be >= 0")
+ return value
+
+
+def _parse_page_size(raw: str | None) -> int:
+ """``page_size`` honours the default when missing, raises when invalid,
+ and silently clamps to ``_MAX_PAGE_SIZE`` (so ``?page_size=500``
+ returns 200 records instead of a 400)."""
+ if raw is None:
+ return _DEFAULT_PAGE_SIZE
+ try:
+ value = int(raw)
+ except (TypeError, ValueError) as exc:
+ raise ActivityParamsError(f"Invalid 'page_size' value: {raw!r}") from exc
+ if value < 1:
+ raise ActivityParamsError("Invalid 'page_size' value: must be >= 1")
+ return min(value, _MAX_PAGE_SIZE)
+
+
+def _parse_iso_datetime(value: str) -> datetime | None:
+ """Parse an ISO-8601 datetime string. Tolerates the trailing ``Z``
+ suffix that Python <3.11 ``fromisoformat`` rejects, and normalises any
+ timezone-aware result to naive UTC.
+
+ The ``since`` / ``until`` filters bind directly against
+ ``version_transaction.issued_at``, which is ``sa.DateTime()`` — a
+ timezone-*naive* column (UTC by convention). Binding a tz-aware value
+ against it shifts the comparison by the session offset on PostgreSQL
+ (and raises on some drivers), so collapse aware inputs to naive UTC
+ here. Naive inputs pass through unchanged (already treated as UTC).
+ """
+ candidate = value[:-1] + "+00:00" if value.endswith("Z") else value
+ try:
+ parsed = datetime.fromisoformat(candidate)
+ except ValueError:
+ return None
+ if parsed.tzinfo is not None:
+ parsed = parsed.astimezone(timezone.utc).replace(tzinfo=None)
+ return parsed
+
+
+def _record_matches(record: dict[str, Any], q: str) -> bool:
+ """Case-insensitive substring match for the ``q`` search filter,
+ over the human-meaningful surfaces of a decorated activity record:
+ ``summary``, ``entity_name``, ``kind``, the joined ``path`` segments,
+ and the JSON form of ``from_value`` / ``to_value`` (JSON, not Python
+ ``str()``: the client searches the serialized text it renders, so
+ ``false`` / ``null`` / double-quoted keys must match — and falsy
+ values like ``False`` / ``0`` must not collapse to unsearchable
+ empty strings).
+ """
+
+ def _value_text(value: Any) -> str:
+ if value is None:
+ return ""
+ try:
+ return json.dumps(value)
+ except (TypeError, ValueError):
+ return str(value)
+
+ needle = q.lower()
+ haystacks = (
+ record.get("summary") or "",
+ record.get("entity_name") or "",
+ record.get("kind") or "",
+ " ".join(str(seg) for seg in (record.get("path") or [])),
+ _value_text(record.get("from_value")),
+ _value_text(record.get("to_value")),
+ )
+ return any(needle in h.lower() for h in haystacks)
+
+
+def get_activity(
+ model_cls: type[Model],
+ entity_uuid: UUID,
+ *,
+ since: datetime | None = None,
+ until: datetime | None = None,
+ include: str = "all",
+ q: str | None = None,
+ page: int = 0,
+ page_size: int = _DEFAULT_PAGE_SIZE,
+ resolved_entity: Any | None = None,
+) -> tuple[list[dict[str, Any]], int, bool]:
+ """Cross-entity activity stream for one path entity.
+
+ Single polymorphic entry point. Dispatches on *model_cls* to
+ assemble the path entity's self records plus the transitive related-
+ entity records (charts attached to a dashboard, datasets a chart
+ pointed at, etc.).
+
+ Returns ``(records, total_count, truncated)``. ``truncated`` is
+ ``True`` when the per-kind fetch ceiling
+ (``queries._MAX_FETCHED_RECORDS``) bit for any kind — the stream is
+ clamped to a clean time cut and older records exist beyond it, so
+ ``count`` is a floor, not the absolute total.
+
+ *resolved_entity*, when supplied, is the already-resolved live path
+ entity (the endpoint resolves it once via ``resolve_endpoint_path_entity``
+ for the ``raise_for_access`` gate); passing it here skips a second
+ identical ``find_active_by_uuid`` lookup and the TOCTOU window between
+ them. The count is post-visibility (silent visibility filter),
+ post-include-filter, and — when ``q`` is supplied — post-
+ search-filter (``count`` reflects the matches, the contract the
+ server-side search exists to provide), not just the size of the
+ returned slice — clients paginate by passing ``page`` forward until
+ ``page * page_size >= count``.
+
+ Raises ``DashboardNotFoundError`` / ``ChartNotFoundError`` /
+ ``DatasetNotFoundError`` when the path entity doesn't exist.
+ """
+ if resolved_entity is not None:
+ path_entity, path_id = resolved_entity, resolved_entity.id
+ else:
+ path_entity, path_id = resolve_path_entity(model_cls, entity_uuid)
+ path_kind = model_cls.__name__
+ kind_key = path_kind.lower() # "dashboard" / "slice" / "sqlatable"
+
+ # Lower-bound the self window at the entity's first tracked transaction
+ # (matched on (id, uuid)) so a reused integer id can't inherit a
+ # hard-deleted predecessor's history.
+ self_start_tx = first_tracked_tx(model_cls, path_id, path_entity.uuid)
+ with _phase_timer(kind_key, "relationship_resolution_ms"):
+ entity_windows = resolve_scope(path_kind, path_id, include, self_start_tx)
+ if not entity_windows:
+ _emit_request_shape_attributes(
+ kind_key,
+ include=include,
+ has_since_filter=since is not None,
+ page_size=page_size,
+ record_count=0,
+ entity_windows=[],
+ path_kind=path_kind,
+ path_id=path_id,
+ )
+ return [], 0, False
+
+ # Visibility filter runs before decoration: it needs the raw
+ # ``entity_id`` column (which decoration strips), and dropping
+ # invisible records early means we don't pay for name lookup +
+ # tombstone probes + impact counts on records the requester
+ # can't see (the silent-filter contract).
+ with _phase_timer(kind_key, "fetch_ms"):
+ records, truncated = fetch_change_records(entity_windows, since, until)
+ with _phase_timer(kind_key, "visibility_filter_ms"):
+ records = filter_records_by_visibility(records)
+ with _phase_timer(kind_key, "denormalize_ms"):
+ apply_entity_name_denormalization(records)
+ # Runs post-visibility (fewer entities to probe) and pre-
+ # decoration (needs the raw table-form entity_kind/entity_id
+ # that decoration rewrites).
+ mark_first_tracked_saves(records)
+ with _phase_timer(kind_key, "decorate_ms"):
+ apply_record_decoration(records, path_kind, path_id)
+
+ # Server-side search (the panel's client-side
+ # search only covers loaded pages). Applied post-decoration so the
+ # synthesized ``summary`` / ``entity_name`` participate, and pre-
+ # count so pagination paginates the MATCHES — the full record set
+ # is already materialized in Python (the documented design),
+ # so the filter adds no extra query.
+ if q:
+ records = [r for r in records if _record_matches(r, q)]
+
+ total = len(records)
+ bounded_size = max(1, min(page_size, _MAX_PAGE_SIZE))
+ offset = max(0, page) * bounded_size
+
+ _emit_request_shape_attributes(
+ kind_key,
+ include=include,
+ has_since_filter=since is not None,
+ page_size=bounded_size,
+ record_count=total,
+ entity_windows=entity_windows,
+ path_kind=path_kind,
+ path_id=path_id,
+ )
+
+ return records[offset : offset + bounded_size], total, truncated
+
+
+def activity_endpoint(
+ api: Any, model_cls: type[Model], uuid_str: str, request_args: Any
+) -> Response:
+ """Body of ``GET /api/v1/{resource}/<uuid>/activity/``.
+
+ Same shape as :func:`superset.versioning.api_helpers.list_versions_endpoint`
+ for the ``/versions/`` endpoint family. Resolves the path entity,
+ parses the request query params, runs :func:`get_activity`, and
+ wraps the result through ``ActivityResponseSchema``.
+
+ *api* is the FAB ``ModelRestApi`` instance (pass ``self`` from the
+ endpoint method). *request_args* is ``request.args`` from
+ ``flask.request`` — passed explicitly so the helper is testable
+ without a live Flask context.
+ """
+ # pylint: disable=import-outside-toplevel
+ from superset.versioning.schemas import ActivityResponseSchema
+
+ try:
+ entity, _ = resolve_endpoint_path_entity(api, model_cls, uuid_str)
+ except PathEntityResponseError as exc:
+ return exc.response
+
+ try:
+ params = parse_activity_query_params(request_args)
+ except ActivityParamsError as exc:
+ return api.response_400(message=str(exc))
+
+ records, count, truncated = get_activity(
+ model_cls, entity.uuid, resolved_entity=entity, **params
+ )
+ payload = ActivityResponseSchema().dump(
+ {"result": records, "count": count, "truncated": truncated}
+ )
+ return api.response(200, **payload)
+
+
+# ---- Observability -------------------------------------------------------
+
+#: Common prefix for every metric this module emits.
+_METRIC_PREFIX = "superset.activity_view"
+
+
+@contextlib.contextmanager
+def _phase_timer(kind_key: str, phase: str) -> Iterator[None]:
+ """Time the wrapped block and emit
+ ``superset.activity_view.<kind>.<phase>`` to ``stats_logger_manager``.
+ Wrapper around :func:`superset.utils.decorators.stats_timing` that
+ centralises the key construction.
+ """
+ # pylint: disable=import-outside-toplevel
+ from superset.extensions import stats_logger_manager
+ from superset.utils.decorators import stats_timing
+
+ with stats_timing(
+ f"{_METRIC_PREFIX}.{kind_key}.{phase}",
+ stats_logger_manager.instance,
+ ):
+ yield
+
+
+def _emit_request_shape_attributes(
+ kind_key: str,
+ *,
+ include: str,
+ has_since_filter: bool,
+ page_size: int,
+ record_count: int,
+ entity_windows: list[EntityWindows],
+ path_kind: str,
+ path_id: int,
+) -> None:
+ """Emit non-PII shape counters about the request and its result set.
+
+ Emits: include_mode / has_since_filter / page_size / record_count
+ + per-related-kind entity counts. **No PII**: entity names, diff
+ content, user identifiers — none of those reach the metric layer.
+ The counters use ``incr`` (counters) since they're tags, not
+ latencies; the timing keys above carry the latency dimension.
+ """
+ # pylint: disable=import-outside-toplevel
+ from superset.extensions import stats_logger_manager
+
+ sl = stats_logger_manager.instance
+
+ # Tag-style metrics: one counter per attribute value. The statsd
+ # bridge accepts arbitrary strings; downstream dashboards filter by
+ # the value segment.
+ sl.incr(f"{_METRIC_PREFIX}.{kind_key}.requests.include_{include}")
+ sl.incr(
+ f"{_METRIC_PREFIX}.{kind_key}.requests."
+ f"has_since_filter_{'true' if has_since_filter else 'false'}"
+ )
+ sl.gauge(f"{_METRIC_PREFIX}.{kind_key}.page_size", float(page_size))
+ sl.gauge(f"{_METRIC_PREFIX}.{kind_key}.record_count", float(record_count))
+
+ # Per-related-kind entity counts. The scope
+ # list includes the path entity itself (the "self" window); exclude
+ # it so the gauge reflects only the *related* entities the request
+ # fanned out to, not "this request touched itself".
+ by_kind: dict[str, int] = {"Slice": 0, "SqlaTable": 0, "Dashboard": 0}
+ for api_kind, entity_id, _windows in entity_windows:
+ if (api_kind, entity_id) == (path_kind, path_id):
+ continue
+ if api_kind in by_kind:
+ by_kind[api_kind] += 1
+ sl.gauge(
+ f"{_METRIC_PREFIX}.{kind_key}.related_entity_count.charts",
+ float(by_kind["Slice"]),
+ )
+ sl.gauge(
+ f"{_METRIC_PREFIX}.{kind_key}.related_entity_count.datasets",
+ float(by_kind["SqlaTable"]),
Review Comment:
**Suggestion:** This metrics block initializes counts for charts, datasets, and dashboards, but only emits gauges for charts and datasets. As a result, dashboard-related fanout is never reported, making request-shape telemetry incomplete and inconsistent with the function’s stated per-kind metrics behavior; emit the missing dashboard gauge as well. [incomplete implementation]
<details>
<summary><b>Severity Level:</b> Major ⚠️</summary>
```mdx
- ⚠️ Activity-view telemetry omits dashboard related-entity count gauge.
- ⚠️ Operators cannot see dashboard fanout in metrics.
```
</details>
<details>
<summary><b>Steps of Reproduction ✅ </b></summary>
```mdx
1. Call any activity endpoint, for example `DashboardRestApi.activity` in
`superset/dashboards/api.py:2614-2623`, which delegates to `activity_endpoint(self,
Dashboard, uuid_str, request.args)` in
`superset/versioning/activity/orchestrator.py:321-355`.
2. Inside `activity_endpoint`, the helper calls `get_activity(model_cls, entity.uuid,
...)` (orchestrator.py:349-351), which in turn invokes
`_emit_request_shape_attributes(...)` at the end of the function (orchestrator.py:307-316)
with the computed `entity_windows` scope.
3. `_emit_request_shape_attributes` in
`superset/versioning/activity/orchestrator.py:382-392` initializes `by_kind` with keys
`"Slice"`, `"SqlaTable"`, and `"Dashboard"` and populates counts from `entity_windows`
while explicitly excluding the self window (lines 421-426), so `by_kind["Dashboard"]` can
be greater than zero when a scope includes related dashboards.
4. The metrics emission block at orchestrator.py:427-433 only calls `sl.gauge` for the
`related_entity_count.charts` and `related_entity_count.datasets` metrics, never emitting
a gauge for dashboards, so dashboard-related fanout counts are computed but never surfaced
in observability; `grep` for `related_entity_count` confirms only charts and datasets
gauges exist (orchestrator.py:428, 432).
```
</details>
[](https://app.codeant.ai/fix-in-ide?tool=cursor&prompt_id=79ff3eec01a942f0888dd1efd2c67d24&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset) [](https://app.codeant.ai/fix-in-ide?tool=vscode-claude&prompt_id=79ff3eec01a942f0888dd1efd2c67d24&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset)
*(Use Cmd/Ctrl + Click for best experience)*
<details>
<summary><b>Prompt for AI Agent 🤖 </b></summary>
```mdx
This is a comment left during a code review.
**Path:** superset/versioning/activity/orchestrator.py
**Line:** 421:433
**Comment:**
*Incomplete Implementation: This metrics block initializes counts for charts, datasets, and dashboards, but only emits gauges for charts and datasets. As a result, dashboard-related fanout is never reported, making request-shape telemetry incomplete and inconsistent with the function’s stated per-kind metrics behavior; emit the missing dashboard gauge as well.
Validate the correctness of the flagged issue. If correct, How can I resolve this? If you propose a fix, implement it and please make it concise.
Once fix is implemented, also check other comments on the same PR, and ask user if the user wants to fix the rest of the comments as well. if said yes, then fetch all the comments validate the correctness and implement a minimal fix
```
</details>
<a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F41076&comment_hash=f012488db504a91bb89540b1b3345bdf00787205ac71b914e89bcdeaa8e6eb85&reaction=like'>👍</a> | <a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F41076&comment_hash=f012488db504a91bb89540b1b3345bdf00787205ac71b914e89bcdeaa8e6eb85&reaction=dislike'>👎</a>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(versioning): cross-entity version activity view [superset]
Posted by "codeant-ai-for-open-source[bot] (via GitHub)" <gi...@apache.org>.
codeant-ai-for-open-source[bot] commented on code in PR #41076:
URL: https://github.com/apache/superset/pull/41076#discussion_r3560247138
##########
superset/dashboards/api.py:
##########
@@ -2540,6 +2539,85 @@ def get_version(self, uuid_str: str, version_uuid_str: str) -> Response:
404:
$ref: '#/components/responses/404'
"""
- return get_version_endpoint(
- self, Dashboard, uuid_str, version_uuid_str, access_kwarg="dashboard"
- )
+ return get_version_endpoint(self, Dashboard, uuid_str, version_uuid_str)
+
+ @expose("/<uuid_str>/activity/", methods=("GET",))
+ @protect()
+ @safe
+ @statsd_metrics
+ @event_logger.log_this_with_context(
+ action=lambda self, *args, **kwargs: f"{self.__class__.__name__}.activity",
+ log_to_statsd=False,
+ )
+ def activity(self, uuid_str: str) -> Response:
Review Comment:
**Suggestion:** This new protected endpoint does not set a permission alias (for example mapping to `get`), so FAB will require `can_activity_Dashboard` instead of the standard read permission. Existing non-admin roles that can read dashboards but do not have this newly named permission will get 403s, effectively breaking access to the endpoint. Add a method-permission mapping (or `@permission_name("get")`) for `activity`. [api mismatch]
<details>
<summary><b>Severity Level:</b> Critical 🚨</summary>
```mdx
❌ Dashboard activity endpoint inaccessible to non-admin dashboard readers.
⚠️ New activity-view UI broken for standard dashboard roles.
```
</details>
<details>
<summary><b>Steps of Reproduction ✅ </b></summary>
```mdx
1. Inspect `DashboardRestApi` in `superset/dashboards/api.py:10-57`; note
`class_permission_name = "Dashboard"` and `method_permission_name` only maps standard CRUD
methods and `"restore"` to `"write"` (lines 45-56), with the comment that missing mappings
make `@protect()` fall back to `can_<method>_<class>` which "standard roles don't carry".
2. Observe the `activity` endpoint definition at `superset/dashboards/api.py:105-113`: it
uses `@expose("/<uuid_str>/activity/", methods=("GET",))` and `@protect()` but has no
`@permission_name("get")` decorator and no `"activity"` entry in `method_permission_name`.
3. Start Superset and log in as a non-admin user whose role grants dashboard read
permissions (can view dashboards via `/api/v1/dashboard/`), but who has not been assigned
any `can_activity_Dashboard` permission (roles are configured from the existing permission
map; no code adds this new permission).
4. Call `GET /api/v1/dashboard/<uuid>/activity/` which is routed to
`DashboardRestApi.activity`; FAB computes the required permission as
`can_activity_Dashboard`, finds that the user lacks it, and `@protect()` denies the
request with HTTP 403, even though the user can read the dashboard itself.
```
</details>
[](https://app.codeant.ai/fix-in-ide?tool=cursor&prompt_id=e328bfb379f84675bb7b7d55899f0d54&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset) [](https://app.codeant.ai/fix-in-ide?tool=vscode-claude&prompt_id=e328bfb379f84675bb7b7d55899f0d54&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset)
*(Use Cmd/Ctrl + Click for best experience)*
<details>
<summary><b>Prompt for AI Agent 🤖 </b></summary>
```mdx
This is a comment left during a code review.
**Path:** superset/dashboards/api.py
**Line:** 2544:2552
**Comment:**
*Api Mismatch: This new protected endpoint does not set a permission alias (for example mapping to `get`), so FAB will require `can_activity_Dashboard` instead of the standard read permission. Existing non-admin roles that can read dashboards but do not have this newly named permission will get 403s, effectively breaking access to the endpoint. Add a method-permission mapping (or `@permission_name("get")`) for `activity`.
Validate the correctness of the flagged issue. If correct, How can I resolve this? If you propose a fix, implement it and please make it concise.
Once fix is implemented, also check other comments on the same PR, and ask user if the user wants to fix the rest of the comments as well. if said yes, then fetch all the comments validate the correctness and implement a minimal fix
```
</details>
<a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F41076&comment_hash=941b559a3c0acb48e42afbe31e13c171b038704047fcecfbfecc1edad539ad6c&reaction=like'>👍</a> | <a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F41076&comment_hash=941b559a3c0acb48e42afbe31e13c171b038704047fcecfbfecc1edad539ad6c&reaction=dislike'>👎</a>
##########
superset/versioning/activity/queries.py:
##########
@@ -0,0 +1,754 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""DB-touching helpers for the activity-view read path.
+
+All Phase A relationship walks (``charts_attached_to_dashboard``,
+``datasets_used_by_chart``, ``batch_datasets_used_by_charts``),
+the Phase B change-record fetch (``fetch_change_records`` /
+``_select_change_rows_for_kinds``), the name-denormalization helpers
+(``_resolve_names_for_kind`` / ``apply_entity_name_denormalization``), the
+path-entity resolution helper (``resolve_path_entity``), and the
+tombstone-state lookup (``check_entity_tombstones``) live here.
+
+Each helper is a thin SELECT-and-shape function — no orchestration,
+no business logic. Callers in :mod:`scope`, :mod:`render`, and
+:mod:`orchestrator` compose them into the end-to-end request.
+
+**Inline imports.** Continuum's ``version_class`` / ``versioning_manager``
+and the Superset model classes are imported inside each helper because
+this package is loaded from ``init_versioning()`` before all SQLAlchemy
+mappers are configured.
+"""
+
+from __future__ import annotations
+
+import logging
+from datetime import datetime
+from typing import Any
+from uuid import UUID
+
+import sqlalchemy as sa
+from flask_appbuilder import Model
+
+from superset.extensions import db
+from superset.versioning.activity.kinds import (
+ API_KIND_TO_TABLE,
+ chunked_ids,
+ EntityWindows,
+ load_live_model,
+ NAME_COLUMN,
+ NOT_FOUND_EXC,
+ TABLE_KIND_TO_API,
+ Window,
+)
+from superset.versioning.activity.windows import row_within_any_window
+from superset.versioning.changes import version_changes_table
+
+logger = logging.getLogger(__name__)
+
+# ---- Path-entity resolution -----------------------------------------------
+
+
+def resolve_path_entity(model_cls: type[Model], entity_uuid: UUID) -> tuple[Any, int]:
+ """Resolve *entity_uuid* to ``(live_entity, entity_id)`` or raise a
+ typed 404.
+
+ Soft-delete handling is inherited transparently from
+ :func:`superset.versioning.queries.find_active_by_uuid` once it
+ learns to filter out ``deleted_at IS NOT NULL`` rows; at that point
+ soft-deleted paths will also raise here.
+ """
+ # pylint: disable=import-outside-toplevel
+ from superset.versioning.queries import find_active_by_uuid
+
+ entity = find_active_by_uuid(model_cls, entity_uuid)
+ if entity is None:
+ api_kind = model_cls.__name__
+ exc_cls = NOT_FOUND_EXC.get(api_kind)
+ if exc_cls is None:
+ raise LookupError(
+ f"Activity view does not support model class {api_kind!r}"
+ )
+ raise exc_cls(str(entity_uuid))
+ return entity, entity.id
+
+
+def first_tracked_tx(
+ model_cls: type[Model], entity_id: int, entity_uuid: UUID
+) -> int | None:
+ """Return the earliest Continuum transaction_id of the shadow rows
+ that belong to *this* live entity, matched on ``(id, uuid)``; ``None``
+ when the entity has no tracked history yet.
+
+ Used to lower-bound the self-scope window. Matching on the integer
+ ``id`` alone would inherit a previously hard-deleted entity's history
+ under id reuse (SQLite/MySQL reuse ``max(id)+1``); pinning the uuid
+ too — the same discrimination :func:`mark_first_tracked_saves` uses —
+ scopes the window to the current entity's own transactions.
+ """
+ # pylint: disable=import-outside-toplevel
+ from sqlalchemy_continuum import version_class
+
+ shadow_tbl = version_class(model_cls).__table__
+ stmt = sa.select(sa.func.min(shadow_tbl.c.transaction_id)).where(
+ shadow_tbl.c.id == entity_id,
+ shadow_tbl.c.uuid == entity_uuid,
+ )
+ return db.session.connection().execute(stmt).scalar()
+
+
+# ---- Phase A: relationship-traversal queries ------------------------------
+
+
+def charts_attached_to_dashboard(dashboard_id: int) -> list[tuple[int, Window]]:
+ """Return ``(slice_id, window)`` for every chart that has ever been on
+ *dashboard_id*, with each association's validity window in
+ transaction-id space.
+
+ Reads from ``dashboard_slices_version`` (Continuum's auto-generated
+ M2M shadow). Rows with ``operation_type = 2`` (DELETE) are excluded
+ so we don't synthesize a phantom window from a detachment row.
+ """
+ # pylint: disable=import-outside-toplevel
+ from sqlalchemy_continuum import version_class
+
+ from superset.models.dashboard import Dashboard
+
+ metadata = version_class(Dashboard).__table__.metadata
+ m2m_tbl = metadata.tables.get("dashboard_slices_version")
+ if m2m_tbl is None:
+ return []
+
+ rows = (
+ db.session.connection()
+ .execute(
+ sa.select(
+ m2m_tbl.c.slice_id,
+ m2m_tbl.c.transaction_id,
+ m2m_tbl.c.end_transaction_id,
+ ).where(
+ m2m_tbl.c.dashboard_id == dashboard_id,
+ m2m_tbl.c.operation_type != 2,
+ m2m_tbl.c.slice_id.is_not(None),
+ )
+ )
+ .all()
+ )
+ result: list[tuple[int, Window]] = []
+ for row in rows:
+ try:
+ window = Window(row[1], row[2])
+ except ValueError:
+ # A degenerate shadow row (end_tx <= start_tx) must not 500 the
+ # endpoint; skip it and leave a breadcrumb for investigation.
+ logger.warning(
+ "activity: skipping degenerate dashboard_slices_version row "
+ "(dashboard_id=%s, slice_id=%s, tx=%s, end_tx=%s)",
+ dashboard_id,
+ row[0],
+ row[1],
+ row[2],
+ )
+ continue
+ result.append((row[0], window))
+ return result
+
+
+def datasets_used_by_chart(slice_id: int) -> list[tuple[int, Window]]:
+ """Return ``(datasource_id, window)`` for every dataset that *slice_id*
+ has ever pointed at, with each association's validity window.
+
+ Single-slice form, used by ``_resolve_chart_scope`` where there
+ is only one chart to walk. The dashboard-scope path calls
+ :func:`batch_datasets_used_by_charts` instead so the query fires
+ once for all slices on the dashboard, not once per slice.
+
+ Reads from ``slices_version`` (the chart parent shadow). Filters to
+ ``datasource_type = 'table'`` because the activity view only follows
+ the chart → ``SqlaTable`` dependency edge (not legacy/other
+ datasources). Rows with ``operation_type = 2`` are excluded.
+ """
+ return batch_datasets_used_by_charts({slice_id}).get(slice_id, [])
+
+
+def batch_datasets_used_by_charts(
+ slice_ids: set[int],
+) -> dict[int, list[tuple[int, Window]]]:
+ """Batch form of :func:`datasets_used_by_chart`. Returns
+ ``{slice_id: [(dataset_id, window), ...]}`` in a single query so the
+ dashboard-scope walker doesn't fire one query per chart on the
+ dashboard. The previous per-slice shape became O(n_charts) round-
+ trips, which dominated ``get_activity`` latency on dashboards with
+ rich history (profile run 2026-05-26 showed `resolve_scope`
+ accounting for ~1.9s out of 4s p95).
+ """
+ if not slice_ids:
+ return {}
+
+ # pylint: disable=import-outside-toplevel
+ from sqlalchemy_continuum import version_class
+
+ from superset.models.slice import Slice
+
+ slices_tbl = version_class(Slice).__table__
+ grouped: dict[int, list[tuple[int, Window]]] = {}
+ # Chunk the IN-clause under SQLite's bind-variable floor (a dashboard can
+ # carry more charts than the floor allows in one statement).
+ rows: list[Any] = []
+ for chunk in chunked_ids(slice_ids):
+ rows.extend(
+ db.session.connection()
+ .execute(
+ sa.select(
+ slices_tbl.c.id,
+ slices_tbl.c.datasource_id,
+ slices_tbl.c.transaction_id,
+ slices_tbl.c.end_transaction_id,
+ ).where(
+ slices_tbl.c.id.in_(chunk),
+ slices_tbl.c.datasource_type == "table",
+ slices_tbl.c.operation_type != 2,
+ slices_tbl.c.datasource_id.is_not(None),
+ )
+ )
+ .mappings()
+ .all()
+ )
+ for row in rows:
+ try:
+ window = Window(row["transaction_id"], row["end_transaction_id"])
+ except ValueError:
+ # A degenerate shadow row (end_tx <= start_tx) must not 500 the
+ # endpoint; skip it and leave a breadcrumb for investigation.
+ logger.warning(
+ "activity: skipping degenerate slices_version row "
+ "(slice_id=%s, datasource_id=%s, tx=%s, end_tx=%s)",
+ row["id"],
+ row["datasource_id"],
+ row["transaction_id"],
+ row["end_transaction_id"],
+ )
+ continue
+ grouped.setdefault(row["id"], []).append((row["datasource_id"], window))
+ return grouped
+
+
+# ---- Phase B: change-record fetch -----------------------------------------
+
+
+def fetch_change_records(
+ entity_window_tuples: list[EntityWindows],
+ since: datetime | None,
+ until: datetime | None,
+) -> tuple[list[dict[str, Any]], bool]:
+ """Fetch all ``version_changes`` rows matching any of the supplied
+ entity-window tuples, joined with ``version_transaction`` for
+ ``issued_at`` and ``user_id``.
+
+ Each tuple is ``(api_kind, entity_id, [(start_tx, end_tx), ...])``;
+ a record matches when ``entity_kind`` equals the table-stored form
+ of *api_kind*, ``entity_id`` matches, and ``transaction_id`` falls
+ inside at least one of the entity's windows. ``since``/``until``
+ further restrict by ``issued_at``.
+
+ Implementation: one SELECT per kind with ``entity_id IN (...)`` and
+ a wide ``transaction_id`` bound (the union of all windows for that
+ kind). Per-window precision is applied in Python afterward. This
+ keeps the SQL shape proportional to the number of *kinds* (≤3) and
+ the bound proportional to the union of windows, not the cross-
+ product of (entity, window) — which previously generated one OR
+ clause per (entity, window) pair and hit SQLite's
+ ``SQLITE_MAX_EXPR_DEPTH`` limit on dashboards with many slices
+ or many historical attachment windows.
+
+ The visibility filter runs after this function (records
+ the requester can't read are silently dropped and must not
+ contribute to ``count``), so the orchestrator paginates in Python
+ over the filtered list — there is no DB-level page ``OFFSET`` here.
+ There IS a per-kind safety ceiling (``_MAX_FETCHED_RECORDS``) that
+ bounds how much history a single request materializes (to at most
+ ``n_kinds * _MAX_FETCHED_RECORDS``); when a kind exhausts it, the
+ second return value is ``True``, the returned stream is clamped to a
+ clean time cut (records older than the highest per-kind fetch floor
+ are dropped so truncation is a time boundary, not per-kind holes), and
+ the caller surfaces ``truncated`` on the response.
+
+ Returns ``(records, truncated)``. Records are ordered by
+ ``(issued_at DESC, transaction_id DESC, sequence DESC)`` — the
+ secondary keys break ties for the stable-ordering contract.
+ """
+ if not entity_window_tuples:
+ return [], False
+
+ # Group windows by (table_kind, entity_id) and by table_kind for SQL
+ # narrowing. The fetch is per-kind; the post-filter is per-entity.
+ windows_by_entity: dict[tuple[str, int], list[Window]] = {}
+ ids_by_kind: dict[str, set[int]] = {}
+ for api_kind, entity_id, windows in entity_window_tuples:
+ table_kind = API_KIND_TO_TABLE.get(api_kind)
+ if table_kind is None or not windows:
+ continue
+ ids_by_kind.setdefault(table_kind, set()).add(entity_id)
+ windows_by_entity.setdefault((table_kind, entity_id), []).extend(windows)
+
+ if not ids_by_kind:
+ return [], False
+
+ # Per-kind transaction_id bounds = the union of that kind's windows.
+ # Pushing these into the SQL WHERE ensures the per-statement LIMIT
+ # selects from IN-WINDOW rows. Without it, a related entity whose
+ # association window is far in the past would have the newest ``limit``
+ # (out-of-window) rows fetched and discarded, silently dropping its
+ # in-window records that lie beyond the limit. ``end_tx = None``
+ # (open-ended/current) means no upper bound for that kind.
+ bounds_by_kind: dict[str, tuple[int, int | None]] = {}
+ for (table_kind, _entity_id), windows in windows_by_entity.items():
+ for w in windows:
+ cur = bounds_by_kind.get(table_kind)
+ if cur is None:
+ bounds_by_kind[table_kind] = (w.start_tx, w.end_tx)
+ continue
+ lo = min(cur[0], w.start_tx)
+ hi = None if (cur[1] is None or w.end_tx is None) else max(cur[1], w.end_tx)
+ bounds_by_kind[table_kind] = (lo, hi)
+
+ rows, truncated, truncation_floor = _select_change_rows_for_kinds(
+ ids_by_kind, bounds_by_kind, since, until, _MAX_FETCHED_RECORDS
+ )
+ filtered = [
+ row
+ for row in rows
+ if row_within_any_window(
+ row, windows_by_entity.get((row["entity_kind"], row["entity_id"]), [])
+ )
+ ]
+ if truncated and truncation_floor is not None:
+ # Collapse truncation to a clean time cut. Below the highest per-kind
+ # fetch floor at least one kind is missing rows, so surfacing the
+ # other kinds there would present an incomplete ("nothing else
+ # changed") picture. Drop everything older than the floor so
+ # ``truncated=True`` means "complete at/after this instant, nothing
+ # shown before it" — a time boundary rather than per-kind holes.
+ filtered = [r for r in filtered if r["issued_at"] >= truncation_floor]
Review Comment:
**Suggestion:** Truncation is collapsed using only `issued_at`, but ordering is actually by `(issued_at, transaction_id, sequence, ...)`. If truncation cuts within a timestamp bucket, this filter still keeps records at that timestamp while older siblings at the same timestamp were dropped by the limit, so the response is not a true “clean cut” and can contain hidden holes. Use a composite floor key matching the full sort tuple instead of timestamp-only comparison. [incorrect condition logic]
<details>
<summary><b>Severity Level:</b> Major ⚠️</summary>
```mdx
⚠️ Truncated activity streams incomplete at boundary timestamp instant.
⚠️ Clients may misinterpret truncated results as fully representative.
```
</details>
<details>
<summary><b>Steps of Reproduction ✅ </b></summary>
```mdx
1. When `get_activity()` in `superset/versioning/activity/orchestrator.py:62-63` calls
`fetch_change_records()` for a scope with very long history,
`_select_change_rows_for_kinds()` (queries.py:365-491) may hit the per-kind `limit`
(`_MAX_FETCHED_RECORDS = 5000` at queries.py:494-501), marking `truncated = True` for that
kind and recording a per-kind `truncation_floor` as `min(r["issued_at"] for r in
kind_rows)` (lines 482-488).
2. `fetch_change_records()` receives `(rows, truncated, truncation_floor)` from
`_select_change_rows_for_kinds()` and builds `filtered` based on exact window membership
(lines 329-338). If `truncated` is true and `truncation_floor` not None, it then executes
`filtered = [r for r in filtered if r["issued_at"] >= truncation_floor]` at
`queries.py:339-346`, using only the timestamp floor to collapse truncation and relying on
the docstring claim that this produces a "clean time cut" (comments at lines 340-345).
3. Final ordering of records is by the composite key `(issued_at, transaction_id,
sequence, entity_kind, entity_id)` descending (queries.py:347-359). When multiple change
records share the same `issued_at` value but some of them were not fetched because the
per-kind limit was reached, the timestamp-only filter keeps the subset of records at that
issued_at that happened to be fetched while omitting other siblings at the same instant,
so the stream is not actually complete "at/after this instant".
4. In truncated responses (for dashboards or charts with >5000 change rows for a kind),
clients can see some records at the boundary timestamp and assume that instant is fully
represented, while other change records that occurred at the same timestamp but were
excluded by the limit remain invisible. Using only `issued_at` in the truncation filter
creates hidden per-kind holes at the boundary instant, contradicting the intended
clean-cut semantics described in the `fetch_change_records()` docstring.
```
</details>
[](https://app.codeant.ai/fix-in-ide?tool=cursor&prompt_id=9720fcd5c4ad4844aeec4a9fd02b09b0&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset) [](https://app.codeant.ai/fix-in-ide?tool=vscode-claude&prompt_id=9720fcd5c4ad4844aeec4a9fd02b09b0&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset)
*(Use Cmd/Ctrl + Click for best experience)*
<details>
<summary><b>Prompt for AI Agent 🤖 </b></summary>
```mdx
This is a comment left during a code review.
**Path:** superset/versioning/activity/queries.py
**Line:** 346:346
**Comment:**
*Incorrect Condition Logic: Truncation is collapsed using only `issued_at`, but ordering is actually by `(issued_at, transaction_id, sequence, ...)`. If truncation cuts within a timestamp bucket, this filter still keeps records at that timestamp while older siblings at the same timestamp were dropped by the limit, so the response is not a true “clean cut” and can contain hidden holes. Use a composite floor key matching the full sort tuple instead of timestamp-only comparison.
Validate the correctness of the flagged issue. If correct, How can I resolve this? If you propose a fix, implement it and please make it concise.
Once fix is implemented, also check other comments on the same PR, and ask user if the user wants to fix the rest of the comments as well. if said yes, then fetch all the comments validate the correctness and implement a minimal fix
```
</details>
<a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F41076&comment_hash=80b71502635fdf1b1d2a0528e41fce4b383cfcb912ab224511faac96d90df19e&reaction=like'>👍</a> | <a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F41076&comment_hash=80b71502635fdf1b1d2a0528e41fce4b383cfcb912ab224511faac96d90df19e&reaction=dislike'>👎</a>
##########
superset/datasets/api.py:
##########
@@ -1808,6 +1807,86 @@ def get_version(self, uuid_str: str, version_uuid_str: str) -> Response:
404:
$ref: '#/components/responses/404'
"""
- return get_version_endpoint(
- self, SqlaTable, uuid_str, version_uuid_str, access_kwarg="datasource"
- )
+ return get_version_endpoint(self, SqlaTable, uuid_str, version_uuid_str)
+
+ @expose("/<uuid_str>/activity/", methods=("GET",))
+ @protect()
+ @safe
+ @statsd_metrics
+ @event_logger.log_this_with_context(
+ action=lambda self, *args, **kwargs: f"{self.__class__.__name__}.activity",
+ log_to_statsd=False,
+ )
+ def activity(self, uuid_str: str) -> Response:
Review Comment:
**Suggestion:** This new protected endpoint also lacks a permission alias to `get`, so FAB falls back to `can_activity_Dataset`/`can_activity_DatasetRestApi`-style permission checks instead of normal read access. Users who can view datasets but were not explicitly granted this new custom permission will receive 403. Map `activity` to `get` (or add `@permission_name("get")`) to keep authorization behavior consistent with other read endpoints. [api mismatch]
<details>
<summary><b>Severity Level:</b> Critical 🚨</summary>
```mdx
❌ Dataset activity endpoint inaccessible to normal dataset readers.
⚠️ Activity history feature unusable without custom role updates.
```
</details>
<details>
<summary><b>Steps of Reproduction ✅ </b></summary>
```mdx
1. Inspect `DatasetRestApi` in `superset/datasets/api.py:1-35`; note
`class_permission_name = "Dataset"` and `method_permission_name` only maps standard CRUD
methods and `"restore"` to `"write"` (lines 7-18), with comments identical to
`DashboardRestApi` about `@protect()` falling back to `can_<method>_<class>` when a
mapping is missing and standard roles not carrying those permissions.
2. Observe the dataset `activity` endpoint at `superset/datasets/api.py:108-116` (diff
lines 1812-1820): it is decorated with `@expose("/<uuid_str>/activity/",
methods=("GET",))`, `@protect()`, `@safe`, `@statsd_metrics`, and logging, but has no
`@permission_name("get")` and `"activity"` is not present in `method_permission_name`.
3. Run Superset and log in as a user whose role allows viewing datasets (can access
`/api/v1/dataset/<id>/` and the dataset list) but who has not been granted a
`can_activity_Dataset` or `can_activity_DatasetRestApi` permission; roles are based on
existing read/write mappings and will not automatically include this new method-specific
permission.
4. Call `GET /api/v1/dataset/<uuid>/activity/`, which invokes `DatasetRestApi.activity`;
FAB resolves the required permission as `can_activity_Dataset` (or
`can_activity_DatasetRestApi`) because no alias is configured, finds it missing on the
role, and `@protect()` returns HTTP 403, preventing normal dataset readers from seeing
their own activity history.
```
</details>
[](https://app.codeant.ai/fix-in-ide?tool=cursor&prompt_id=de9d0f78f1e7449eb9d8cabb4022cbf8&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset) [](https://app.codeant.ai/fix-in-ide?tool=vscode-claude&prompt_id=de9d0f78f1e7449eb9d8cabb4022cbf8&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset)
*(Use Cmd/Ctrl + Click for best experience)*
<details>
<summary><b>Prompt for AI Agent 🤖 </b></summary>
```mdx
This is a comment left during a code review.
**Path:** superset/datasets/api.py
**Line:** 1812:1820
**Comment:**
*Api Mismatch: This new protected endpoint also lacks a permission alias to `get`, so FAB falls back to `can_activity_Dataset`/`can_activity_DatasetRestApi`-style permission checks instead of normal read access. Users who can view datasets but were not explicitly granted this new custom permission will receive 403. Map `activity` to `get` (or add `@permission_name("get")`) to keep authorization behavior consistent with other read endpoints.
Validate the correctness of the flagged issue. If correct, How can I resolve this? If you propose a fix, implement it and please make it concise.
Once fix is implemented, also check other comments on the same PR, and ask user if the user wants to fix the rest of the comments as well. if said yes, then fetch all the comments validate the correctness and implement a minimal fix
```
</details>
<a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F41076&comment_hash=15c4bccb52cecab4b102e9e0e1300bdcc131c39d32ec9e0abab1065d0b9c848c&reaction=like'>👍</a> | <a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F41076&comment_hash=15c4bccb52cecab4b102e9e0e1300bdcc131c39d32ec9e0abab1065d0b9c848c&reaction=dislike'>👎</a>
##########
superset/versioning/activity/queries.py:
##########
@@ -0,0 +1,754 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""DB-touching helpers for the activity-view read path.
+
+All Phase A relationship walks (``charts_attached_to_dashboard``,
+``datasets_used_by_chart``, ``batch_datasets_used_by_charts``),
+the Phase B change-record fetch (``fetch_change_records`` /
+``_select_change_rows_for_kinds``), the name-denormalization helpers
+(``_resolve_names_for_kind`` / ``apply_entity_name_denormalization``), the
+path-entity resolution helper (``resolve_path_entity``), and the
+tombstone-state lookup (``check_entity_tombstones``) live here.
+
+Each helper is a thin SELECT-and-shape function — no orchestration,
+no business logic. Callers in :mod:`scope`, :mod:`render`, and
+:mod:`orchestrator` compose them into the end-to-end request.
+
+**Inline imports.** Continuum's ``version_class`` / ``versioning_manager``
+and the Superset model classes are imported inside each helper because
+this package is loaded from ``init_versioning()`` before all SQLAlchemy
+mappers are configured.
+"""
+
+from __future__ import annotations
+
+import logging
+from datetime import datetime
+from typing import Any
+from uuid import UUID
+
+import sqlalchemy as sa
+from flask_appbuilder import Model
+
+from superset.extensions import db
+from superset.versioning.activity.kinds import (
+ API_KIND_TO_TABLE,
+ chunked_ids,
+ EntityWindows,
+ load_live_model,
+ NAME_COLUMN,
+ NOT_FOUND_EXC,
+ TABLE_KIND_TO_API,
+ Window,
+)
+from superset.versioning.activity.windows import row_within_any_window
+from superset.versioning.changes import version_changes_table
+
+logger = logging.getLogger(__name__)
+
+# ---- Path-entity resolution -----------------------------------------------
+
+
+def resolve_path_entity(model_cls: type[Model], entity_uuid: UUID) -> tuple[Any, int]:
+ """Resolve *entity_uuid* to ``(live_entity, entity_id)`` or raise a
+ typed 404.
+
+ Soft-delete handling is inherited transparently from
+ :func:`superset.versioning.queries.find_active_by_uuid` once it
+ learns to filter out ``deleted_at IS NOT NULL`` rows; at that point
+ soft-deleted paths will also raise here.
+ """
+ # pylint: disable=import-outside-toplevel
+ from superset.versioning.queries import find_active_by_uuid
+
+ entity = find_active_by_uuid(model_cls, entity_uuid)
+ if entity is None:
+ api_kind = model_cls.__name__
+ exc_cls = NOT_FOUND_EXC.get(api_kind)
+ if exc_cls is None:
+ raise LookupError(
+ f"Activity view does not support model class {api_kind!r}"
+ )
+ raise exc_cls(str(entity_uuid))
+ return entity, entity.id
+
+
+def first_tracked_tx(
+ model_cls: type[Model], entity_id: int, entity_uuid: UUID
+) -> int | None:
+ """Return the earliest Continuum transaction_id of the shadow rows
+ that belong to *this* live entity, matched on ``(id, uuid)``; ``None``
+ when the entity has no tracked history yet.
+
+ Used to lower-bound the self-scope window. Matching on the integer
+ ``id`` alone would inherit a previously hard-deleted entity's history
+ under id reuse (SQLite/MySQL reuse ``max(id)+1``); pinning the uuid
+ too — the same discrimination :func:`mark_first_tracked_saves` uses —
+ scopes the window to the current entity's own transactions.
+ """
+ # pylint: disable=import-outside-toplevel
+ from sqlalchemy_continuum import version_class
+
+ shadow_tbl = version_class(model_cls).__table__
+ stmt = sa.select(sa.func.min(shadow_tbl.c.transaction_id)).where(
+ shadow_tbl.c.id == entity_id,
+ shadow_tbl.c.uuid == entity_uuid,
+ )
+ return db.session.connection().execute(stmt).scalar()
+
+
+# ---- Phase A: relationship-traversal queries ------------------------------
+
+
+def charts_attached_to_dashboard(dashboard_id: int) -> list[tuple[int, Window]]:
+ """Return ``(slice_id, window)`` for every chart that has ever been on
+ *dashboard_id*, with each association's validity window in
+ transaction-id space.
+
+ Reads from ``dashboard_slices_version`` (Continuum's auto-generated
+ M2M shadow). Rows with ``operation_type = 2`` (DELETE) are excluded
+ so we don't synthesize a phantom window from a detachment row.
+ """
+ # pylint: disable=import-outside-toplevel
+ from sqlalchemy_continuum import version_class
+
+ from superset.models.dashboard import Dashboard
+
+ metadata = version_class(Dashboard).__table__.metadata
+ m2m_tbl = metadata.tables.get("dashboard_slices_version")
+ if m2m_tbl is None:
+ return []
+
+ rows = (
+ db.session.connection()
+ .execute(
+ sa.select(
+ m2m_tbl.c.slice_id,
+ m2m_tbl.c.transaction_id,
+ m2m_tbl.c.end_transaction_id,
+ ).where(
+ m2m_tbl.c.dashboard_id == dashboard_id,
+ m2m_tbl.c.operation_type != 2,
+ m2m_tbl.c.slice_id.is_not(None),
+ )
+ )
+ .all()
+ )
+ result: list[tuple[int, Window]] = []
+ for row in rows:
+ try:
+ window = Window(row[1], row[2])
+ except ValueError:
+ # A degenerate shadow row (end_tx <= start_tx) must not 500 the
+ # endpoint; skip it and leave a breadcrumb for investigation.
+ logger.warning(
+ "activity: skipping degenerate dashboard_slices_version row "
+ "(dashboard_id=%s, slice_id=%s, tx=%s, end_tx=%s)",
+ dashboard_id,
+ row[0],
+ row[1],
+ row[2],
+ )
+ continue
+ result.append((row[0], window))
+ return result
+
+
+def datasets_used_by_chart(slice_id: int) -> list[tuple[int, Window]]:
+ """Return ``(datasource_id, window)`` for every dataset that *slice_id*
+ has ever pointed at, with each association's validity window.
+
+ Single-slice form, used by ``_resolve_chart_scope`` where there
+ is only one chart to walk. The dashboard-scope path calls
+ :func:`batch_datasets_used_by_charts` instead so the query fires
+ once for all slices on the dashboard, not once per slice.
+
+ Reads from ``slices_version`` (the chart parent shadow). Filters to
+ ``datasource_type = 'table'`` because the activity view only follows
+ the chart → ``SqlaTable`` dependency edge (not legacy/other
+ datasources). Rows with ``operation_type = 2`` are excluded.
+ """
+ return batch_datasets_used_by_charts({slice_id}).get(slice_id, [])
+
+
+def batch_datasets_used_by_charts(
+ slice_ids: set[int],
+) -> dict[int, list[tuple[int, Window]]]:
+ """Batch form of :func:`datasets_used_by_chart`. Returns
+ ``{slice_id: [(dataset_id, window), ...]}`` in a single query so the
+ dashboard-scope walker doesn't fire one query per chart on the
+ dashboard. The previous per-slice shape became O(n_charts) round-
+ trips, which dominated ``get_activity`` latency on dashboards with
+ rich history (profile run 2026-05-26 showed `resolve_scope`
+ accounting for ~1.9s out of 4s p95).
+ """
+ if not slice_ids:
+ return {}
+
+ # pylint: disable=import-outside-toplevel
+ from sqlalchemy_continuum import version_class
+
+ from superset.models.slice import Slice
+
+ slices_tbl = version_class(Slice).__table__
+ grouped: dict[int, list[tuple[int, Window]]] = {}
+ # Chunk the IN-clause under SQLite's bind-variable floor (a dashboard can
+ # carry more charts than the floor allows in one statement).
+ rows: list[Any] = []
+ for chunk in chunked_ids(slice_ids):
+ rows.extend(
+ db.session.connection()
+ .execute(
+ sa.select(
+ slices_tbl.c.id,
+ slices_tbl.c.datasource_id,
+ slices_tbl.c.transaction_id,
+ slices_tbl.c.end_transaction_id,
+ ).where(
+ slices_tbl.c.id.in_(chunk),
+ slices_tbl.c.datasource_type == "table",
+ slices_tbl.c.operation_type != 2,
+ slices_tbl.c.datasource_id.is_not(None),
+ )
+ )
+ .mappings()
+ .all()
+ )
+ for row in rows:
+ try:
+ window = Window(row["transaction_id"], row["end_transaction_id"])
+ except ValueError:
+ # A degenerate shadow row (end_tx <= start_tx) must not 500 the
+ # endpoint; skip it and leave a breadcrumb for investigation.
+ logger.warning(
+ "activity: skipping degenerate slices_version row "
+ "(slice_id=%s, datasource_id=%s, tx=%s, end_tx=%s)",
+ row["id"],
+ row["datasource_id"],
+ row["transaction_id"],
+ row["end_transaction_id"],
+ )
+ continue
+ grouped.setdefault(row["id"], []).append((row["datasource_id"], window))
+ return grouped
+
+
+# ---- Phase B: change-record fetch -----------------------------------------
+
+
+def fetch_change_records(
+ entity_window_tuples: list[EntityWindows],
+ since: datetime | None,
+ until: datetime | None,
+) -> tuple[list[dict[str, Any]], bool]:
+ """Fetch all ``version_changes`` rows matching any of the supplied
+ entity-window tuples, joined with ``version_transaction`` for
+ ``issued_at`` and ``user_id``.
+
+ Each tuple is ``(api_kind, entity_id, [(start_tx, end_tx), ...])``;
+ a record matches when ``entity_kind`` equals the table-stored form
+ of *api_kind*, ``entity_id`` matches, and ``transaction_id`` falls
+ inside at least one of the entity's windows. ``since``/``until``
+ further restrict by ``issued_at``.
+
+ Implementation: one SELECT per kind with ``entity_id IN (...)`` and
+ a wide ``transaction_id`` bound (the union of all windows for that
+ kind). Per-window precision is applied in Python afterward. This
+ keeps the SQL shape proportional to the number of *kinds* (≤3) and
+ the bound proportional to the union of windows, not the cross-
+ product of (entity, window) — which previously generated one OR
+ clause per (entity, window) pair and hit SQLite's
+ ``SQLITE_MAX_EXPR_DEPTH`` limit on dashboards with many slices
+ or many historical attachment windows.
+
+ The visibility filter runs after this function (records
+ the requester can't read are silently dropped and must not
+ contribute to ``count``), so the orchestrator paginates in Python
+ over the filtered list — there is no DB-level page ``OFFSET`` here.
+ There IS a per-kind safety ceiling (``_MAX_FETCHED_RECORDS``) that
+ bounds how much history a single request materializes (to at most
+ ``n_kinds * _MAX_FETCHED_RECORDS``); when a kind exhausts it, the
+ second return value is ``True``, the returned stream is clamped to a
+ clean time cut (records older than the highest per-kind fetch floor
+ are dropped so truncation is a time boundary, not per-kind holes), and
+ the caller surfaces ``truncated`` on the response.
+
+ Returns ``(records, truncated)``. Records are ordered by
+ ``(issued_at DESC, transaction_id DESC, sequence DESC)`` — the
+ secondary keys break ties for the stable-ordering contract.
+ """
+ if not entity_window_tuples:
+ return [], False
+
+ # Group windows by (table_kind, entity_id) and by table_kind for SQL
+ # narrowing. The fetch is per-kind; the post-filter is per-entity.
+ windows_by_entity: dict[tuple[str, int], list[Window]] = {}
+ ids_by_kind: dict[str, set[int]] = {}
+ for api_kind, entity_id, windows in entity_window_tuples:
+ table_kind = API_KIND_TO_TABLE.get(api_kind)
+ if table_kind is None or not windows:
+ continue
+ ids_by_kind.setdefault(table_kind, set()).add(entity_id)
+ windows_by_entity.setdefault((table_kind, entity_id), []).extend(windows)
+
+ if not ids_by_kind:
+ return [], False
+
+ # Per-kind transaction_id bounds = the union of that kind's windows.
+ # Pushing these into the SQL WHERE ensures the per-statement LIMIT
+ # selects from IN-WINDOW rows. Without it, a related entity whose
+ # association window is far in the past would have the newest ``limit``
+ # (out-of-window) rows fetched and discarded, silently dropping its
+ # in-window records that lie beyond the limit. ``end_tx = None``
+ # (open-ended/current) means no upper bound for that kind.
+ bounds_by_kind: dict[str, tuple[int, int | None]] = {}
+ for (table_kind, _entity_id), windows in windows_by_entity.items():
+ for w in windows:
+ cur = bounds_by_kind.get(table_kind)
+ if cur is None:
+ bounds_by_kind[table_kind] = (w.start_tx, w.end_tx)
+ continue
+ lo = min(cur[0], w.start_tx)
+ hi = None if (cur[1] is None or w.end_tx is None) else max(cur[1], w.end_tx)
+ bounds_by_kind[table_kind] = (lo, hi)
+
+ rows, truncated, truncation_floor = _select_change_rows_for_kinds(
+ ids_by_kind, bounds_by_kind, since, until, _MAX_FETCHED_RECORDS
+ )
+ filtered = [
+ row
+ for row in rows
+ if row_within_any_window(
+ row, windows_by_entity.get((row["entity_kind"], row["entity_id"]), [])
+ )
+ ]
+ if truncated and truncation_floor is not None:
+ # Collapse truncation to a clean time cut. Below the highest per-kind
+ # fetch floor at least one kind is missing rows, so surfacing the
+ # other kinds there would present an incomplete ("nothing else
+ # changed") picture. Drop everything older than the floor so
+ # ``truncated=True`` means "complete at/after this instant, nothing
+ # shown before it" — a time boundary rather than per-kind holes.
+ filtered = [r for r in filtered if r["issued_at"] >= truncation_floor]
+ # Sort key must be TOTAL so pagination is stable across requests: two
+ # records from different entities can share (issued_at, transaction_id,
+ # sequence), so append (entity_kind, entity_id) to break remaining ties
+ # deterministically. Without these the relative order of tied records
+ # depends on set-iteration order and a record could shift pages.
+ filtered.sort(
+ key=lambda r: (
+ r["issued_at"],
+ r["transaction_id"],
+ r["sequence"],
+ r["entity_kind"],
+ r["entity_id"],
+ ),
+ reverse=True,
+ )
+ return filtered, truncated
+
+
+def _select_change_rows_for_kinds(
+ ids_by_kind: dict[str, set[int]],
+ bounds_by_kind: dict[str, tuple[int, int | None]],
+ since: datetime | None,
+ until: datetime | None,
+ limit: int,
+) -> tuple[list[dict[str, Any]], bool, datetime | None]:
+ """Fire one SELECT per entity_kind with ``entity_id IN (...)``;
+ concatenate the results. Each SELECT joins ``version_transaction``
+ + ``ab_user`` so the orchestrator has the columns it needs for
+ decoration.
+
+ Per-kind, not one query: SQLAlchemy's ``tuple_(entity_kind,
+ entity_id).in_(...)`` would collapse the three queries into one,
+ but its SQL emission is not portable across Postgres, MySQL, and
+ SQLite. The per-kind shape is the correct trade-off given
+ Superset's multi-dialect requirement (at most 3 round-trips per
+ request, bounded by the kind taxonomy). Do not "optimise" into a
+ composite-tuple IN clause without verifying the SQL on all three
+ dialects.
+
+ **Init-order dependency.** ``tx_tbl.c.action_kind`` resolves only
+ after ``init_versioning()`` has run — the column is appended onto
+ Continuum's transaction Table by
+ ``superset.versioning.factory.VersionTransactionFactory`` at app
+ start via ``append_column`` + ``add_property``. This helper is
+ safe to call from request-path code because the app is fully
+ initialised by then; calling it from a script that imports the
+ versioning package without going through ``init_versioning()``
+ will raise ``AttributeError`` on the ``action_kind`` attribute
+ access below."""
+ # pylint: disable=import-outside-toplevel
+ from sqlalchemy_continuum import versioning_manager
+
+ from superset import security_manager
+
+ tx_tbl = versioning_manager.transaction_cls.__table__
+ user_tbl = security_manager.user_model.__table__
+ vc = version_changes_table
+ join_tree = vc.join(tx_tbl, vc.c.transaction_id == tx_tbl.c.id).outerjoin(
+ user_tbl, tx_tbl.c.user_id == user_tbl.c.id
+ )
+ select_cols = (
+ vc.c.transaction_id,
+ vc.c.entity_kind,
+ vc.c.entity_id,
+ vc.c.sequence,
+ vc.c.kind,
+ vc.c.operation,
+ vc.c.path,
+ vc.c.from_value,
+ vc.c.to_value,
+ tx_tbl.c.issued_at,
+ tx_tbl.c.user_id,
+ # ``action_kind`` is the high-level avenue (restore / import /
+ # clone / NULL=ordinary save) stamped by the originating
+ # command via the change-record listener. All records sharing a
+ # ``transaction_id`` share the same value. The column is
+ # declared on the Continuum Table by ``VersionTransactionFactory``,
+ # so ``tx_tbl.c.action_kind`` resolves cleanly here. See
+ # the three change-record dimensions.
+ tx_tbl.c.action_kind,
+ user_tbl.c.id.label("changed_by_id"),
+ user_tbl.c.first_name,
+ user_tbl.c.last_name,
+ )
+
+ out: list[dict[str, Any]] = []
+ truncated = False
+ truncation_floors: list[datetime] = []
+ for table_kind, entity_ids in ids_by_kind.items():
+ # The fetch budget is per KIND, not per id-chunk. A kind whose
+ # entity_ids span several bind-variable chunks shares one ``limit``
+ # across those chunks, so the rows a single request materializes are
+ # bounded by ``n_kinds * limit`` (<= 3 * _MAX_FETCHED_RECORDS), not
+ # ``n_chunks * limit``. ``entity_ids`` is chunked to stay inside
+ # SQLite's ``SQLITE_MAX_VARIABLE_NUMBER`` floor (default 999 in many
+ # builds); Postgres + MySQL accept the full list but the chunk is
+ # dialect-agnostic for simplicity.
+ kind_rows: list[dict[str, Any]] = []
+ for chunk in chunked_ids(entity_ids):
+ remaining = limit - len(kind_rows)
+ if remaining <= 0:
+ break
+ stmt = (
+ sa.select(*select_cols)
+ .select_from(join_tree)
+ .where(
+ vc.c.entity_kind == table_kind,
+ vc.c.entity_id.in_(chunk),
+ )
+ )
+ # Bound by the kind's window union so the LIMIT picks in-window
+ # rows (see fetch_change_records). The per-entity window filter
+ # still runs in Python afterwards for exact membership.
+ tx_lo, tx_hi = bounds_by_kind[table_kind]
+ stmt = stmt.where(vc.c.transaction_id >= tx_lo)
+ if tx_hi is not None:
+ stmt = stmt.where(vc.c.transaction_id < tx_hi)
+ if since is not None:
+ stmt = stmt.where(tx_tbl.c.issued_at >= since)
+ if until is not None:
+ stmt = stmt.where(tx_tbl.c.issued_at < until)
+ # Match fetch_change_records' final Python sort key order
+ # (issued_at, transaction_id, sequence, entity_id) so the
+ # per-kind budget keeps exactly the rows the final sort ranks
+ # highest. entity_kind is constant within a per-kind statement.
+ stmt = stmt.order_by(
+ tx_tbl.c.issued_at.desc(),
+ vc.c.transaction_id.desc(),
+ vc.c.sequence.desc(),
+ vc.c.entity_id.desc(),
+ ).limit(remaining)
+ kind_rows.extend(
+ dict(row)
+ for row in db.session.connection().execute(stmt).mappings().all()
+ )
Review Comment:
**Suggestion:** The per-kind fetch limit is applied while iterating ID chunks, which means once early chunks fill `remaining`, later chunks are never queried. Because `entity_ids` comes from a set and chunk order is non-deterministic, this can silently drop newer records from skipped chunks and make results unstable across requests for large scopes (> chunk size). Fetch each chunk independently and apply the limit only after globally merging/sorting all chunk rows for that kind. [logic error]
<details>
<summary><b>Severity Level:</b> Major ⚠️</summary>
```mdx
❌ Newest activity records can be omitted for large dashboards.
⚠️ Activity results unstable when exceeding per-kind fetch limit.
```
</details>
<details>
<summary><b>Steps of Reproduction ✅ </b></summary>
```mdx
1. In `superset/versioning/activity/orchestrator.py:38-44`, `get_activity()` calls
`resolve_scope()` to build `entity_windows` for the path entity (dashboard, chart, or
dataset), then in `orchestrator.py:62-63` calls `fetch_change_records(entity_windows,
since, until)` from `superset/versioning/activity/queries.py` for every `/activity/`
request.
2. Inside `fetch_change_records()` (`queries.py:253-331`), the code groups windows by
`(table_kind, entity_id)` and then calls `_select_change_rows_for_kinds(ids_by_kind,
bounds_by_kind, since, until, _MAX_FETCHED_RECORDS)` with `limit = _MAX_FETCHED_RECORDS`
(5000, defined at `queries.py:494-501`) as a per-kind ceiling on fetched change rows.
3. `_select_change_rows_for_kinds()` (`queries.py:365-491`) builds `ids_by_kind` values as
`set[int]` for each kind (line 366-367), then iterates `for table_kind, entity_ids in
ids_by_kind.items():` and, for each kind, chunk-splits `entity_ids` via
`chunked_ids(entity_ids)` (line 445), which converts the set to a list without any
deterministic sort in `superset/versioning/activity/kinds.py:54-62`.
4. For each chunk, the query at `queries.py:449-477` orders rows by `(issued_at DESC,
transaction_id DESC, sequence DESC, entity_id DESC)` and applies `.limit(remaining)` where
`remaining = limit - len(kind_rows)` (lines 446-447). As soon as earlier chunks
collectively fill `kind_rows` to `limit`, later chunks are skipped (`if remaining <= 0:
break` at line 447), so if some entity IDs happen to fall into later chunks and have
change rows newer than many rows from earlier chunks, their records are silently dropped.
Because `entity_ids` is a set and chunk grouping depends on arbitrary set iteration order,
large scopes (>5000 records for a kind) can return activity streams missing some of the
globally newest records and the exact subset depends on chunk ordering.
```
</details>
[](https://app.codeant.ai/fix-in-ide?tool=cursor&prompt_id=c304b16de7cd48edab1afc003ebcd610&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset) [](https://app.codeant.ai/fix-in-ide?tool=vscode-claude&prompt_id=c304b16de7cd48edab1afc003ebcd610&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset)
*(Use Cmd/Ctrl + Click for best experience)*
<details>
<summary><b>Prompt for AI Agent 🤖 </b></summary>
```mdx
This is a comment left during a code review.
**Path:** superset/versioning/activity/queries.py
**Line:** 444:481
**Comment:**
*Logic Error: The per-kind fetch limit is applied while iterating ID chunks, which means once early chunks fill `remaining`, later chunks are never queried. Because `entity_ids` comes from a set and chunk order is non-deterministic, this can silently drop newer records from skipped chunks and make results unstable across requests for large scopes (> chunk size). Fetch each chunk independently and apply the limit only after globally merging/sorting all chunk rows for that kind.
Validate the correctness of the flagged issue. If correct, How can I resolve this? If you propose a fix, implement it and please make it concise.
Once fix is implemented, also check other comments on the same PR, and ask user if the user wants to fix the rest of the comments as well. if said yes, then fetch all the comments validate the correctness and implement a minimal fix
```
</details>
<a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F41076&comment_hash=3d1e50c9ee30986e88a6178c21179e68b9b3d2116495f747f53225d3855acfc2&reaction=like'>👍</a> | <a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F41076&comment_hash=3d1e50c9ee30986e88a6178c21179e68b9b3d2116495f747f53225d3855acfc2&reaction=dislike'>👎</a>
##########
superset/versioning/activity/render.py:
##########
@@ -0,0 +1,268 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Decoration: turn raw change-records into the ActivityRecord DTO.
+
+After fetching + filtering, each record needs the synthesized fields
+the API contract documents — ``entity_kind`` translated to the user-
+facing form, ``entity_uuid``, ``entity_deleted`` /
+``entity_deletion_state``, ``source`` (self vs. related),
+``summary`` (the headline), ``impact`` (chart-count for
+dashboard→dataset records), ``version_uuid``, ``changed_by``.
+
+This module collects all those decorations:
+
+* :func:`apply_record_decoration` — orchestrates the per-page additions in
+ one pass: pulls tombstones + uuids + impact counts in batches, then
+ walks records adding the synthesized fields and stripping the
+ internal-only columns the API contract doesn't expose.
+* :func:`_lookup_entity_uuids` — one IN-clause query per kind to
+ resolve live ``uuid`` for non-tombstoned entities.
+* :func:`_build_summary` — pure projection of (api_kind, record kind,
+ entity_name) onto the headline string.
+* :func:`_changed_by_dict` — projects the user columns onto the
+ ``changed_by`` DTO shape.
+"""
+
+from __future__ import annotations
+
+from typing import Any
+from uuid import UUID
+
+import sqlalchemy as sa
+
+from superset.extensions import db
+from superset.versioning.activity.impact import (
+ batch_chart_counts,
+ collect_impact_pairs,
+ impact_for_record,
+)
+from superset.versioning.activity.kinds import (
+ API_KIND_LABEL,
+ chunked_ids,
+ load_live_model,
+ NAME_COLUMN,
+ TABLE_KIND_TO_API,
+ USER_FACING_KIND,
+)
+from superset.versioning.activity.queries import check_entity_tombstones
+from superset.versioning.queries import derive_version_uuid
+
+_SUMMARY_VERBS: dict[str, str] = {
+ # The kind taxonomy mapped to past-tense verbs for the
+ # "<entity_label> <verb>: <entity_name>" headline. "field" is
+ # the fallback for scalar changes that don't map to a named verb.
+ "filter": "filter changed",
+ "metric": "metric changed",
+ "dimension": "dimension changed",
+ "column": "column changed",
+ "chart": "chart changed",
+ "time_range": "time range changed",
+ "color_palette": "palette changed",
+ "restore": "restored",
+ "field": "updated",
+}
+
+
+def apply_record_decoration(
+ records: list[dict[str, Any]],
+ path_kind: str,
+ path_id: int,
+) -> None:
+ """Add the synthesized ActivityRecord fields to each record in place:
+ ``entity_kind`` (translated to API form), ``entity_uuid``,
+ ``entity_deleted``, ``entity_deletion_state``, ``source``,
+ ``summary``, ``impact``, ``version_uuid``, ``changed_by``.
+
+ Mutates *records* in place; returns ``None``. Records are expected
+ to already carry ``entity_name`` from
+ :func:`apply_entity_name_denormalization`. The in-place mutation
+ avoids re-allocating thousands of dicts on hot dashboards; the
+ name + return signature make the side effect explicit instead of
+ pretending to be a pure projection.
+ """
+ if not records:
+ return
+
+ distinct: set[tuple[str, int]] = {
+ (
+ TABLE_KIND_TO_API.get(r["entity_kind"], ""),
+ r["entity_id"],
+ )
+ for r in records
+ if TABLE_KIND_TO_API.get(r["entity_kind"])
+ }
+ tombstones = check_entity_tombstones(distinct)
+ uuids = _lookup_entity_uuids(distinct, tombstones)
+ # Pre-compute impact counts for the whole page in one batch query
+ # instead of one COUNT per related record (was N+1).
+ impact_counts = batch_chart_counts(
+ path_id, collect_impact_pairs(records, path_kind)
+ )
+
+ for record in records:
+ api_kind = TABLE_KIND_TO_API.get(record["entity_kind"], "")
+ entity_id = record["entity_id"]
+ tombstone = tombstones.get(
+ (api_kind, entity_id), {"deleted": True, "deletion_state": None}
+ )
+ entity_uuid = uuids.get((api_kind, entity_id))
+ is_self = api_kind == path_kind and entity_id == path_id
+
+ # Emit the user-facing form ("dashboard"/"chart"/"dataset") on the
+ # wire; the internal class-name (api_kind) is kept above for the
+ # remaining decoration steps that key off model_cls.__name__.
+ record["entity_kind"] = USER_FACING_KIND.get(api_kind, api_kind)
+ record["entity_uuid"] = str(entity_uuid) if entity_uuid else None
+ record["entity_deleted"] = tombstone["deleted"]
+ record["entity_deletion_state"] = tombstone["deletion_state"]
+ record["source"] = "self" if is_self else "related"
+ record["version_uuid"] = (
+ str(derive_version_uuid(entity_uuid, record["transaction_id"]))
+ if entity_uuid
+ else None
+ )
+ record["changed_by"] = _changed_by_dict(record)
+
+ if is_self:
+ # Self records are left summary-less (the panel renders
+ # them from kind/path/values) — EXCEPT synthetic ``__meta__``
+ # headlines, whose entire payload IS the summary and whose
+ # primary surface is the entity's own stream ("restored to
+ # version N" must render on include=self).
+ record["summary"] = (
+ _build_summary(api_kind, record)
+ if record.get("kind") == "__meta__"
+ else ""
+ )
+ record["impact"] = None
+ else:
+ record["summary"] = _build_summary(api_kind, record)
+ record["impact"] = impact_for_record(record, path_kind, impact_counts)
+ if record["entity_deleted"]:
+ # Security: a tombstoned related entity has no live row, so
+ # the visibility filter cannot access-gate it (there is
+ # nothing to apply the FAB access filter to). Deletion must
+ # not WIDEN what a requester entitled only to the path entity
+ # can see, so redact everything that would identify the
+ # deleted related entity or its editors: its name, the
+ # synthesized headline, the editor identity, and the raw diff
+ # CONTENT (filter values, column names, SQL/adhoc
+ # expressions). The record still appears — kind, operation,
+ # and timestamps remain, rendered as a generic
+ # "(deleted) <kind>" marker — so the stream stays honest
+ # about WHEN something changed without disclosing WHAT, WHO,
+ # or WHICH entity. Self-path tombstones are untouched: the
+ # endpoint already gated them via ``raise_for_access`` on the
+ # path entity.
+ label = API_KIND_LABEL.get(api_kind, api_kind)
+ record["entity_name"] = ""
+ record["summary"] = f"(deleted) {label}"
+ record["changed_by"] = None
+ record["from_value"] = None
+ record["to_value"] = None
+ record["path"] = None
+
+ # Strip the internal-only columns the API contract doesn't expose.
+ for key in (
+ "entity_id",
+ "sequence",
+ "user_id",
+ "changed_by_id",
+ "first_name",
+ "last_name",
+ ):
+ record.pop(key, None)
+
+
+def _lookup_entity_uuids(
+ distinct: set[tuple[str, int]],
+ tombstones: dict[tuple[str, int], dict[str, Any]],
+) -> dict[tuple[str, int], UUID | None]:
+ """Batch-fetch live ``uuid`` per ``(api_kind, entity_id)``. Tombstoned
+ entities are skipped (their ``entity_uuid`` is null).
+ """
+ result: dict[tuple[str, int], UUID | None] = {}
+ by_kind: dict[str, list[int]] = {}
+ for api_kind, entity_id in distinct:
+ if tombstones.get((api_kind, entity_id), {}).get("deleted"):
+ continue
+ by_kind.setdefault(api_kind, []).append(entity_id)
+
+ # ``no_autoflush`` mirrors the defensive posture of the baseline +
+ # change-record listeners: this helper reads from live tables to
+ # resolve uuids, and a future caller that resolves an entity before
+ # the parent flush would otherwise trigger autoflush mid-read.
+ # Today's call sites run from request-path code with no pending
+ # session state, so the cost of the guard is zero.
+ with db.session.no_autoflush:
+ for api_kind, entity_ids in by_kind.items():
+ if api_kind not in NAME_COLUMN:
+ continue
+ model_cls = load_live_model(NAME_COLUMN[api_kind][0])
+ live_tbl = model_cls.__table__
+ # Chunk the IN-clause to stay under SQLite's bind-variable floor
+ # (a wide dashboard can have more related entities than the floor).
+ for chunk in chunked_ids(entity_ids):
+ rows = (
+ db.session.connection()
+ .execute(
+ sa.select(live_tbl.c.id, live_tbl.c.uuid).where(
+ live_tbl.c.id.in_(chunk)
+ )
+ )
+ .all()
+ )
+ for row in rows:
+ result[(api_kind, row[0])] = row[1]
Review Comment:
**Suggestion:** The UUID decoration is resolved only from the current live row by integer `id`, then reused for every historical record of that `entity_id`. If the database reuses IDs after hard delete (already called out elsewhere in this module family), older records from the predecessor entity will be decorated with the new entity’s UUID and a wrong `version_uuid`. Resolve UUID per record transaction window (or join through version/shadow rows using `transaction_id`) instead of a single live-row lookup by `id`. [stale reference]
<details>
<summary><b>Severity Level:</b> Major ⚠️</summary>
```mdx
❌ Activity records show wrong entity_uuid for reused IDs.
⚠️ Derived version_uuid cannot resolve predecessors' historic snapshots.
⚠️ Cross-entity activity stream misidentifies related historical changes.
```
</details>
<details>
<summary><b>Steps of Reproduction ✅ </b></summary>
```mdx
1. Run Superset with this PR on a database engine that reuses integer primary keys after
hard delete (explicitly mentioned in the ID-reuse comment on `first_tracked_tx` in
`superset/versioning/activity/queries.py:90-102`, which notes SQLite/MySQL reuse
`max(id)+1` and warns about inheriting a hard-deleted predecessor's history when matching
only on `id`).
2. Create an entity (e.g., a dataset or chart) and perform edits so Continuum tracks
version rows and `version_changes` records for its `entity_id`; then hard-delete that
entity and create a new entity of the same kind that reuses the same integer `id` (so
historical `version_changes.entity_id` rows now refer to a different live UUID).
3. Call an activity endpoint, for example `GET /api/v1/dashboard/<uuid>/activity/`
implemented by `DashboardRestApi.activity` in `superset/dashboards/api.py:2552-2581`,
which delegates to `activity_endpoint` (`superset/dashboards/api.py:2618-2623`).
`activity_endpoint` (`superset/versioning/activity/orchestrator.py:321-35`) resolves the
path entity via `resolve_endpoint_path_entity`
(`superset/versioning/api_helpers.py:7-59`), parses query params, and calls `get_activity`
(`superset/versioning/activity/orchestrator.py:211-52`).
4. Inside `get_activity`, the helper builds `(api_kind, entity_id, windows)` tuples using
`resolve_scope` (`superset/versioning/activity/scope.py:47-74`) and fetches matching
change rows with `fetch_change_records`
(`superset/versioning/activity/queries.py:253-40`), which matches `version_changes` rows
solely on `(entity_kind, entity_id)` plus transaction window (queries.py:10-13). When
decoration runs, `apply_record_decoration`
(`superset/versioning/activity/render.py:80-139`) gathers a distinct set of `(api_kind,
entity_id)` and passes it to `_lookup_entity_uuids` (`render.py:191-231).
`_lookup_entity_uuids` performs a live-table lookup by integer `id` only
(`render.py:219-230`), returning the current entity's UUID and applying it to **all**
historical records for that `entity_id`. Older records that belong to the hard-deleted
predecessor entity are therefore decorated with the successor entity's `entity_uuid`, and
`derive_version_uuid` (`superset/versioning/queries.py:56-65`) computes a `version_uuid`
for them using the wrong `entity_uuid`, so those activity entries carry an incorrect
entity identity and an invalid version UUID.
```
</details>
[](https://app.codeant.ai/fix-in-ide?tool=cursor&prompt_id=5cd7d86842944d1cbe89404f8ec5b3e1&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset) [](https://app.codeant.ai/fix-in-ide?tool=vscode-claude&prompt_id=5cd7d86842944d1cbe89404f8ec5b3e1&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset)
*(Use Cmd/Ctrl + Click for best experience)*
<details>
<summary><b>Prompt for AI Agent 🤖 </b></summary>
```mdx
This is a comment left during a code review.
**Path:** superset/versioning/activity/render.py
**Line:** 219:230
**Comment:**
*Stale Reference: The UUID decoration is resolved only from the current live row by integer `id`, then reused for every historical record of that `entity_id`. If the database reuses IDs after hard delete (already called out elsewhere in this module family), older records from the predecessor entity will be decorated with the new entity’s UUID and a wrong `version_uuid`. Resolve UUID per record transaction window (or join through version/shadow rows using `transaction_id`) instead of a single live-row lookup by `id`.
Validate the correctness of the flagged issue. If correct, How can I resolve this? If you propose a fix, implement it and please make it concise.
Once fix is implemented, also check other comments on the same PR, and ask user if the user wants to fix the rest of the comments as well. if said yes, then fetch all the comments validate the correctness and implement a minimal fix
```
</details>
<a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F41076&comment_hash=29eba36f07b7597ea56dfb6b840e1bde6c65eb9daba9fb7df2121027464cb4d3&reaction=like'>👍</a> | <a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F41076&comment_hash=29eba36f07b7597ea56dfb6b840e1bde6c65eb9daba9fb7df2121027464cb4d3&reaction=dislike'>👎</a>
##########
superset/versioning/activity/scope.py:
##########
@@ -0,0 +1,118 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Scope resolution — turn a path entity into the related-entity walk.
+
+Composes :mod:`~superset.versioning.activity.queries` (Phase A
+relationship walks) and :mod:`~superset.versioning.activity.windows`
+(pure interval arithmetic) into the
+``list[EntityWindows]`` scope that
+:func:`~superset.versioning.activity.queries.fetch_change_records`
+consumes.
+
+The functions here read the DB (via the Phase A helpers in
+:mod:`~superset.versioning.activity.queries`); the pure window-
+arithmetic functions previously colocated here now live in
+:mod:`~superset.versioning.activity.windows` so the package no longer
+needs a lazy import to dodge a ``scope ↔ queries`` cycle.
+"""
+
+from __future__ import annotations
+
+from superset.versioning.activity.kinds import EntityWindows, Window
+from superset.versioning.activity.queries import (
+ batch_datasets_used_by_charts,
+ charts_attached_to_dashboard,
+ datasets_used_by_chart,
+)
+from superset.versioning.activity.windows import (
+ intersect_windows,
+ merge_entity_windows,
+)
+
+
+def resolve_scope(
+ path_kind: str,
+ path_id: int,
+ include: str,
+ self_start_tx: int | None = None,
+) -> list[EntityWindows]:
+ """Build the ``[(api_kind, entity_id, [windows])]`` list that
+ :func:`~superset.versioning.activity.queries.fetch_change_records`
+ consumes, branching by *path_kind* and *include* mode.
+
+ *self_start_tx* lower-bounds the self window at the path entity's first
+ tracked transaction (see
+ :func:`~superset.versioning.activity.queries.first_tracked_tx`). This
+ scopes the self stream to the current entity's own history: matching
+ self records on the bare integer id would otherwise inherit a
+ hard-deleted predecessor's change records under id reuse
+ (SQLite/MySQL reuse ``max(id)+1``). ``None`` means the entity has no
+ tracked history yet, so the self stream contributes nothing.
+ """
+ want_self = include in ("all", "self")
+ want_related = include in ("all", "related")
+
+ scope: list[EntityWindows] = []
+ if want_self and self_start_tx is not None:
+ scope.append((path_kind, path_id, [Window(self_start_tx, None)]))
+ if want_related:
+ scope.extend(_resolve_related_scope(path_kind, path_id))
+ return scope
+
+
+def _resolve_related_scope(path_kind: str, path_id: int) -> list[EntityWindows]:
+ """Walk the dependency edges from the path entity to its related
+ entities. Datasets have no transitive layer in V2."""
+ if path_kind == "Dashboard":
+ return _resolve_dashboard_scope(path_id)
+ if path_kind == "Slice":
+ return _resolve_chart_scope(path_id)
+ return []
+
+
+def _resolve_dashboard_scope(dashboard_id: int) -> list[EntityWindows]:
+ """Charts on the dashboard during their attachment window, plus
+ datasets each chart pointed at during the intersection of (chart-
+ attachment, chart-on-dataset)."""
+ scope: list[EntityWindows] = []
+ chart_windows: dict[int, list[Window]] = {}
+ for slice_id, window in charts_attached_to_dashboard(dashboard_id):
+ chart_windows.setdefault(slice_id, []).append(window)
+
+ # One query for the dataset-history of every chart on the dashboard,
+ # not one query per chart. The per-slice form was O(n_charts) round-
+ # trips which dominated p95 on rich dashboards.
+ dataset_windows_by_slice = batch_datasets_used_by_charts(set(chart_windows))
+
+ for slice_id, attachment_windows in chart_windows.items():
+ scope.append(("Slice", slice_id, list(attachment_windows)))
+ dataset_windows = dataset_windows_by_slice.get(slice_id, [])
+ for attachment in attachment_windows:
+ for dataset_id, chart_dataset_window in dataset_windows:
+ if (
+ intersect := intersect_windows(attachment, chart_dataset_window)
+ ) is not None:
+ scope.append(("SqlaTable", dataset_id, [intersect]))
Review Comment:
**Suggestion:** Related-entity scope is built from relationship/history rows keyed only by integer IDs, without any UUID disambiguation like the self-scope `self_start_tx` guard. Under ID reuse, this can pull predecessor chart/dataset history into a different current entity’s activity stream. Add UUID-aware scoping (or transaction lower bounds tied to the current related entity identity) for related entities before emitting windows. [stale reference]
<details>
<summary><b>Severity Level:</b> Major ⚠️</summary>
```mdx
❌ Dashboard activity stream includes predecessor dashboard relationships.
⚠️ Chart activity stream can inherit unrelated dataset history.
⚠️ Activity view misleads about changes for current entity.
```
</details>
<details>
<summary><b>Steps of Reproduction ✅ </b></summary>
```mdx
1. On a deployment using a database where integer IDs can be reused after hard delete
(captured in the ID-reuse warning in `first_tracked_tx` at
`superset/versioning/activity/queries.py:90-102`), create dashboard A with id `N`, attach
charts and datasets so relationship history is recorded in Continuum’s shadow tables, then
hard-delete dashboard A and later create dashboard B that reuses integer id `N`.
2. Invoke the dashboard activity endpoint `GET /api/v1/dashboard/<uuid>/activity/`, whose
handler `DashboardRestApi.activity` in `superset/dashboards/api.py:2552-2581` delegates to
the shared `activity_endpoint` helper (`superset/dashboards/api.py:2618-2623`).
`activity_endpoint` (`superset/versioning/activity/orchestrator.py:321-35`) resolves the
path entity via `resolve_endpoint_path_entity`, parses query params, and calls
`get_activity` (`orchestrator.py:211-52`).
3. Within `get_activity`, Superset lowers the self window using
`first_tracked_tx(model_cls, path_id, path_entity.uuid)` (queries.py:90-102) so the
self-stream for dashboard B does not inherit dashboard A’s own history, then calls
`resolve_scope(path_kind, path_id, include, self_start_tx)`
(`superset/versioning/activity/scope.py:47-74). However, `_resolve_dashboard_scope`
(`scope.py:87-110`) builds related-entity windows purely from integer IDs: it calls
`charts_attached_to_dashboard(dashboard_id)` to read dashboard–chart attachment windows
(`superset/versioning/activity/queries.py:117-168`) and
`batch_datasets_used_by_charts(set(chart_windows))` for chart–dataset windows
(`queries.py:188-247`), never applying `self_start_tx` or any UUID discrimination to clip
these windows to dashboard B’s lifetime.
4. Because the shadow tables `dashboard_slices_version` and `slices_version` are keyed
only by `dashboard_id` and `slice_id`, the windows returned for id `N` cover attachment
history for both dashboards A and B when IDs are reused. `merge_entity_windows`
(`superset/versioning/activity/windows.py:59-80`) merges all windows per `(api_kind,
entity_id)` tuple, and `fetch_change_records` (`queries.py:253-40`) matches
`version_changes` rows whenever `(entity_kind, entity_id, transaction_id)` fall inside any
of those windows. The result is that dashboard B’s `/activity/` stream includes related
chart/dataset change records that actually belong to dashboard A’s historical
relationships. A similar pattern applies to chart activity:
`_resolve_chart_scope(slice_id)` (`scope.py:113-118) uses
`datasets_used_by_chart(slice_id)` (queries.py:171-185) keyed only on integer `slice_id`,
so a chart B reusing an ID can inherit predecessor chart A’s dataset history into its
`/activity/` stream.
```
</details>
[](https://app.codeant.ai/fix-in-ide?tool=cursor&prompt_id=b0d45729e90c460fb2c3c090d67922b2&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset) [](https://app.codeant.ai/fix-in-ide?tool=vscode-claude&prompt_id=b0d45729e90c460fb2c3c090d67922b2&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset)
*(Use Cmd/Ctrl + Click for best experience)*
<details>
<summary><b>Prompt for AI Agent 🤖 </b></summary>
```mdx
This is a comment left during a code review.
**Path:** superset/versioning/activity/scope.py
**Line:** 93:109
**Comment:**
*Stale Reference: Related-entity scope is built from relationship/history rows keyed only by integer IDs, without any UUID disambiguation like the self-scope `self_start_tx` guard. Under ID reuse, this can pull predecessor chart/dataset history into a different current entity’s activity stream. Add UUID-aware scoping (or transaction lower bounds tied to the current related entity identity) for related entities before emitting windows.
Validate the correctness of the flagged issue. If correct, How can I resolve this? If you propose a fix, implement it and please make it concise.
Once fix is implemented, also check other comments on the same PR, and ask user if the user wants to fix the rest of the comments as well. if said yes, then fetch all the comments validate the correctness and implement a minimal fix
```
</details>
<a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F41076&comment_hash=22e7c0a544debfa6c3dfa0179fd405db6dde05bbc6994acdd077c7d07cd654df&reaction=like'>👍</a> | <a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F41076&comment_hash=22e7c0a544debfa6c3dfa0179fd405db6dde05bbc6994acdd077c7d07cd654df&reaction=dislike'>👎</a>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(versioning): cross-entity version activity view [superset]
Posted by "bito-code-review[bot] (via GitHub)" <gi...@apache.org>.
bito-code-review[bot] commented on PR #41076:
URL: https://github.com/apache/superset/pull/41076#issuecomment-4835820240
AI Code Review is in progress (usually takes 3 to 15 minutes unless it's a very large PR).
<details>
<summary>
Bito Usage Guide
</summary>
**Commands**
Type the following command in the pull request comment and save the comment.
- `/review` - Manually triggers a full AI review.
- `/pause` - Pauses automatic reviews on this pull request.
- `/resume` - Resumes automatic reviews.
- `/resolve` - Marks all Bito-posted review comments as resolved.
- `/abort` - Cancels all in-progress reviews.
Refer to the <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/available-commands" target="_blank" rel="noopener noreferrer">documentation</a> for additional commands.
**Configuration**
This repository uses `Superset` You can customize the agent settings <a href="https://alpha.bito.ai/home/ai-agents/code-review-agent" target="_blank" rel="noopener noreferrer">here</a> or contact your Bito workspace admin at evan@preset.io.
**Documentation & Help**
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/getting-started/install-run-using-bito-cloud/create-or-customize-an-agent-instance" target="_blank" rel="noopener noreferrer">Customize agent settings</a>
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/implementing-custom-code-review-rules" target="_blank" rel="noopener noreferrer">Review rules</a>
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent" target="_blank" rel="noopener noreferrer">General documentation</a>
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/faqs" target="_blank" rel="noopener noreferrer">FAQ</a>
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(versioning): cross-entity version activity view [superset]
Posted by "mikebridge (via GitHub)" <gi...@apache.org>.
mikebridge commented on PR #41076:
URL: https://github.com/apache/superset/pull/41076#issuecomment-4937282803
🤖 This review was generated via Codex using parallel Preset, Python, React, and Superset committer review passes.
Thanks for the substantial work on the cross-entity activity view. The implementation and test coverage are extensive, but I found two correctness issues that should be resolved before merging.
### Blocking findings
**1. The per-kind cap can omit newer activity across ID chunks**
In `superset/versioning/activity/queries.py:445`, each ID chunk is queried using only the budget left by earlier chunks. Because the IDs originate from sets, chunk order is not a meaningful global ordering.
An earlier chunk can consume the 5,000-row cap even when a later chunk contains newer activity. The resulting stream can therefore return older records while silently omitting newer ones.
Could we instead fetch a bounded candidate set from every chunk, merge the candidates using the complete ordering key, and retain the newest global `limit` rows? Fetching or probing one additional row would also let us set `truncated` only when more data actually exists.
A regression test with more than 500 related entities, where a later chunk contains the newest record, would help lock this down.
**2. Related history is not bounded to the current entity incarnation**
In `superset/versioning/activity/scope.py:73`, `self_start_tx` protects records for the entity itself, but related traversal can still read relationship and entity-version rows associated with a previously deleted object that used the same integer ID.
If the database reuses an ID, the newly created dashboard or chart could display activity belonging to its predecessor.
Could we propagate the lower transaction boundary for the current entity into related-scope resolution and clip the relationship windows accordingly? Matching shadow records against the current UUID where available would provide additional protection.
### Minor documentation points
- `PathEntityResponseError` and `resolve_endpoint_path_entity` in `superset/versioning/api_helpers.py:143` should begin with complete, single-line PEP 257 summaries.
- The dated profiling reference in `superset/versioning/activity/queries.py:196` should use timeless wording while preserving the useful performance rationale.
Overall, the architecture looks thoughtfully developed, but the chunk-ordering and entity-incarnation issues can affect the correctness of the activity stream and should be addressed before merge.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(versioning): cross-entity version activity view [superset]
Posted by "codeant-ai-for-open-source[bot] (via GitHub)" <gi...@apache.org>.
codeant-ai-for-open-source[bot] commented on code in PR #41076:
URL: https://github.com/apache/superset/pull/41076#discussion_r3500171841
##########
superset/datasets/api.py:
##########
@@ -440,17 +504,69 @@ def put(self, pk: int) -> Response:
# This validates custom Schema with custom validations
except ValidationError as error:
return self.response_400(message=error.messages)
+
+ # Live version identifiers before the update (empty + query-free when
+ # ``ENABLE_VERSIONING_CAPTURE`` is off).
+ old_info = current_entity_version_info(SqlaTable, pk)
+
try:
+ # Two commands, two commits, two Continuum transactions for an
+ # ``override_columns`` save — deliberately NOT merged into one
+ # transaction. A single-transaction design was attempted and
+ # reverted: ``DBEventLogger`` writes request logs through the
+ # SHARED scoped session and calls ``commit()`` /
+ # ``rollback()`` on it mid-request (superset/utils/log.py),
+ # so any save held uncommitted across a logged sub-action can
+ # be committed half-done (Postgres/MySQL) or rolled back
+ # entirely on a transient logger failure (SQLite's
+ # "database is locked"). Until the event logger gets its own
+ # session, per-command commit boundaries are the only shape
+ # whose failure modes are honest. Consequence the
+ # version-history UI must tolerate: one logical save can
+ # surface as two version transactions stamped the same second.
changed_model = UpdateDatasetCommand(pk, item, override_columns).run()
+ # Capture the post-update identifiers BEFORE the refresh:
+ # RefreshDatasetCommand commits its own transaction, so reading
+ # afterwards would attribute the refresh's version to the
+ # user's update (and old→new would span two transactions).
+ new_info = current_entity_version_info(
+ SqlaTable, changed_model.id, changed_model.uuid
+ )
+ etag_version_uuid = new_info.version_uuid
if override_columns:
RefreshDatasetCommand(pk).run()
- response = self.response(200, id=changed_model.id, result=item)
+ # The ETag must reflect the entity's *current live* version,
+ # which after the refresh is the refresh's transaction —
+ # re-read it rather than reusing the pre-refresh uuid.
+ etag_version_uuid = current_entity_etag_uuid(
+ SqlaTable, changed_model.id, changed_model.uuid
+ )
+ response = self.response(
+ 200,
+ id=changed_model.id,
+ result=item,
+ old_version=old_info.version,
+ new_version=new_info.version,
+ old_transaction_id=old_info.transaction_id,
+ new_transaction_id=new_info.transaction_id,
+ old_version_uuid=old_info.version_uuid,
+ new_version_uuid=new_info.version_uuid,
+ )
+ set_version_etag(response, etag_version_uuid)
except DatasetNotFoundError:
response = self.response_404()
Review Comment:
**Suggestion:** The second command (`RefreshDatasetCommand`) re-runs ownership validation after the update has already committed; if the update changed owners and removed the caller, refresh can now fail with forbidden even though the dataset update already succeeded. This yields a 403 response for a request that actually mutated state. Run refresh without a second ownership gate for this internal flow, or perform both actions under a consistent authorization model before committing. [logic error]
<details>
<summary><b>Severity Level:</b> Major ⚠️</summary>
```mdx
- ❌ Dataset update persists but responds forbidden to caller.
- ⚠️ UI may misreport failed save despite applied changes.
```
</details>
<details>
<summary><b>Steps of Reproduction ✅ </b></summary>
```mdx
1. As an existing owner of a dataset, issue `PUT
/api/v1/dataset/<pk>?override_columns=true` to `DatasetRestApi.put()` in
`superset/datasets/api.py:3-179` with a request body that changes the `owners` field so
the caller will no longer be listed as an owner after the update.
2. Inside `UpdateDatasetCommand.validate()`
(`superset/commands/dataset/update.py:98-128`),
`security_manager.raise_for_ownership(self._model)` at lines 107-111 checks ownership
against the pre-update state, authorises the caller, and the subsequent
`DatasetDAO.update()` call at lines 93-97 commits the new owners (removing the caller) in
the first transaction.
3. After this commit, `DatasetRestApi.put()` computes `new_info =
current_entity_version_info(SqlaTable, changed_model.id, changed_model.uuid)` at
`superset/datasets/api.py:552-555`; because `override_columns` is true, it then calls
`RefreshDatasetCommand(pk).run()` at lines 556-557, whose `validate()` method
(`superset/commands/dataset/refresh.py:68-77`) re-fetches the dataset and re-runs
`security_manager.raise_for_ownership(self._model)` against the now-updated owners list.
4. Since the caller removed themselves from `owners`, this second ownership check raises
`DatasetForbiddenError` (`superset/commands/dataset/exceptions.py:185-187`), which is
propagated back to `DatasetRestApi.put()` and caught in the `except DatasetForbiddenError`
block at `superset/datasets/api.py:159-160`, causing a 403 error response even though the
dataset update from `UpdateDatasetCommand` has already been successfully committed to the
database.
```
</details>
[](https://app.codeant.ai/fix-in-ide?tool=cursor&prompt_id=4d4a963e461f4dddbb10bff6ce41ae63&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset) [](https://app.codeant.ai/fix-in-ide?tool=vscode-claude&prompt_id=4d4a963e461f4dddbb10bff6ce41ae63&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset)
*(Use Cmd/Ctrl + Click for best experience)*
<details>
<summary><b>Prompt for AI Agent 🤖 </b></summary>
```mdx
This is a comment left during a code review.
**Path:** superset/datasets/api.py
**Line:** 556:557
**Comment:**
*Logic Error: The second command (`RefreshDatasetCommand`) re-runs ownership validation after the update has already committed; if the update changed owners and removed the caller, refresh can now fail with forbidden even though the dataset update already succeeded. This yields a 403 response for a request that actually mutated state. Run refresh without a second ownership gate for this internal flow, or perform both actions under a consistent authorization model before committing.
Validate the correctness of the flagged issue. If correct, How can I resolve this? If you propose a fix, implement it and please make it concise.
Once fix is implemented, also check other comments on the same PR, and ask user if the user wants to fix the rest of the comments as well. if said yes, then fetch all the comments validate the correctness and implement a minimal fix
```
</details>
<a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F41076&comment_hash=e5a3564866b260be3a8b316cbeb1044de2bb57d445282d05db44e5a288b255c3&reaction=like'>👍</a> | <a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F41076&comment_hash=e5a3564866b260be3a8b316cbeb1044de2bb57d445282d05db44e5a288b255c3&reaction=dislike'>👎</a>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(versioning): cross-entity version activity view [superset]
Posted by "codeant-ai-for-open-source[bot] (via GitHub)" <gi...@apache.org>.
codeant-ai-for-open-source[bot] commented on code in PR #41076:
URL: https://github.com/apache/superset/pull/41076#discussion_r3500130747
##########
superset/daos/dataset.py:
##########
@@ -275,6 +275,103 @@ def update(
return super().update(item, attributes)
+ @classmethod
+ def _validate_column_date_formats(
+ cls, property_columns: list[dict[str, Any]]
+ ) -> None:
+ for column in property_columns:
+ if column.get("python_date_format") is None:
+ continue
+ if not DatasetDAO.validate_python_date_format(column["python_date_format"]):
+ raise ValueError(
+ "python_date_format is an invalid date/timestamp format."
+ )
+
+ @classmethod
+ def _override_columns(
+ cls, model: SqlaTable, property_columns: list[dict[str, Any]]
+ ) -> None:
+ """Replace columns by natural key (``column_name``) — update in place
+ rather than delete-and-reinsert.
+
+ SPIKE (full-Continuum): the previous
+ delete-and-reinsert pattern produced overlapping shadow rows in
+ ``table_columns_version`` (the same ``column_name`` had a DELETE
+ shadow at tx N alongside an INSERT shadow at tx N for a fresh PK).
+ Continuum's ``Reverter`` couldn't unwind this on restore: its flush
+ ordering inserts the historical row before deleting the live one,
+ hitting the ``UNIQUE (table_id, column_name)`` constraint mid-flush
+ (ADR-004 Failure 1).
+
+ The natural-key upsert keeps PKs stable across metadata refresh.
+ Continuum captures only real field changes; new columns get plain
+ INSERT shadows; removed columns get plain DELETE shadows. No
+ natural-key collisions, so Reverter can restore cleanly.
+
+ Behaviour change vs. the previous implementation: PKs of unchanged
+ columns are preserved. Charts that reference columns by their
+ ``id`` continue to work across a metadata refresh — previously
+ such references would be invalidated.
+ """
+ existing_by_name = {c.column_name: c for c in model.columns}
Review Comment:
**Suggestion:** Add an explicit type annotation for this newly introduced lookup map to satisfy the type-hinting rule for relevant variables. [custom_rule]
**Severity Level:** Minor ⚠️
<details>
<summary><b>Why it matters? 🤔 </b></summary>
This is a newly added local variable in modified Python code, and it is an annotatable lookup map without an explicit type hint, so it matches the type-hint rule.
</details>
[](https://app.codeant.ai/fix-in-ide?tool=cursor&prompt_id=13a59401e58c46c69691c2f5760a8ff6&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset) [](https://app.codeant.ai/fix-in-ide?tool=vscode-claude&prompt_id=13a59401e58c46c69691c2f5760a8ff6&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset)
*(Use Cmd/Ctrl + Click for best experience)*
<details>
<summary><b>Prompt for AI Agent 🤖 </b></summary>
```mdx
This is a comment left during a code review.
**Path:** superset/daos/dataset.py
**Line:** 316:316
**Comment:**
*Custom Rule: Add an explicit type annotation for this newly introduced lookup map to satisfy the type-hinting rule for relevant variables.
Validate the correctness of the flagged issue. If correct, How can I resolve this? If you propose a fix, implement it and please make it concise.
Once fix is implemented, also check other comments on the same PR, and ask user if the user wants to fix the rest of the comments as well. if said yes, then fetch all the comments validate the correctness and implement a minimal fix
```
</details>
<a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F41076&comment_hash=b9ea6687955bd858c3018845e376d36e28afa86524492c94b23bbab4baab710c&reaction=like'>👍</a> | <a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F41076&comment_hash=b9ea6687955bd858c3018845e376d36e28afa86524492c94b23bbab4baab710c&reaction=dislike'>👎</a>
##########
superset/daos/dataset.py:
##########
@@ -275,6 +275,103 @@ def update(
return super().update(item, attributes)
+ @classmethod
+ def _validate_column_date_formats(
+ cls, property_columns: list[dict[str, Any]]
+ ) -> None:
+ for column in property_columns:
+ if column.get("python_date_format") is None:
+ continue
+ if not DatasetDAO.validate_python_date_format(column["python_date_format"]):
+ raise ValueError(
+ "python_date_format is an invalid date/timestamp format."
+ )
+
+ @classmethod
+ def _override_columns(
+ cls, model: SqlaTable, property_columns: list[dict[str, Any]]
+ ) -> None:
+ """Replace columns by natural key (``column_name``) — update in place
+ rather than delete-and-reinsert.
+
+ SPIKE (full-Continuum): the previous
+ delete-and-reinsert pattern produced overlapping shadow rows in
+ ``table_columns_version`` (the same ``column_name`` had a DELETE
+ shadow at tx N alongside an INSERT shadow at tx N for a fresh PK).
+ Continuum's ``Reverter`` couldn't unwind this on restore: its flush
+ ordering inserts the historical row before deleting the live one,
+ hitting the ``UNIQUE (table_id, column_name)`` constraint mid-flush
+ (ADR-004 Failure 1).
+
+ The natural-key upsert keeps PKs stable across metadata refresh.
+ Continuum captures only real field changes; new columns get plain
+ INSERT shadows; removed columns get plain DELETE shadows. No
+ natural-key collisions, so Reverter can restore cleanly.
+
+ Behaviour change vs. the previous implementation: PKs of unchanged
+ columns are preserved. Charts that reference columns by their
+ ``id`` continue to work across a metadata refresh — previously
+ such references would be invalidated.
+ """
+ existing_by_name = {c.column_name: c for c in model.columns}
+ incoming_by_name = {p["column_name"]: p for p in property_columns}
+
+ # Identity is the natural key here, never the payload's ``id``:
+ # setattr-ing an incoming ``id`` onto a name-matched row would
+ # rewrite a live primary key, and a renamed column whose payload
+ # still carries its old ``id`` would INSERT with a live PK while
+ # the old-named row is deleted in the same flush — INSERTs flush
+ # before DELETEs, so that collides on the PK / UNIQUE(table_id,
+ # column_name) constraints. ``table_id`` is pinned to *model*.
+ protected_keys = ("id", "table_id")
Review Comment:
**Suggestion:** Add a type annotation for this new tuple constant so its expected element type is explicit. [custom_rule]
**Severity Level:** Minor ⚠️
<details>
<summary><b>Why it matters? 🤔 </b></summary>
This is a new tuple constant in modified code and it lacks an explicit type annotation, which is exactly what the rule flags for relevant variables.
</details>
[](https://app.codeant.ai/fix-in-ide?tool=cursor&prompt_id=edbde84e52134cbe8327371a11f3c364&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset) [](https://app.codeant.ai/fix-in-ide?tool=vscode-claude&prompt_id=edbde84e52134cbe8327371a11f3c364&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset)
*(Use Cmd/Ctrl + Click for best experience)*
<details>
<summary><b>Prompt for AI Agent 🤖 </b></summary>
```mdx
This is a comment left during a code review.
**Path:** superset/daos/dataset.py
**Line:** 326:326
**Comment:**
*Custom Rule: Add a type annotation for this new tuple constant so its expected element type is explicit.
Validate the correctness of the flagged issue. If correct, How can I resolve this? If you propose a fix, implement it and please make it concise.
Once fix is implemented, also check other comments on the same PR, and ask user if the user wants to fix the rest of the comments as well. if said yes, then fetch all the comments validate the correctness and implement a minimal fix
```
</details>
<a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F41076&comment_hash=b21128a5ede41b69612339875d56c13155248805014b721f89bf5e94fe3c0485&reaction=like'>👍</a> | <a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F41076&comment_hash=b21128a5ede41b69612339875d56c13155248805014b721f89bf5e94fe3c0485&reaction=dislike'>👎</a>
##########
superset/daos/dataset.py:
##########
@@ -275,6 +275,103 @@ def update(
return super().update(item, attributes)
+ @classmethod
+ def _validate_column_date_formats(
+ cls, property_columns: list[dict[str, Any]]
+ ) -> None:
+ for column in property_columns:
+ if column.get("python_date_format") is None:
+ continue
+ if not DatasetDAO.validate_python_date_format(column["python_date_format"]):
+ raise ValueError(
+ "python_date_format is an invalid date/timestamp format."
+ )
+
+ @classmethod
+ def _override_columns(
+ cls, model: SqlaTable, property_columns: list[dict[str, Any]]
+ ) -> None:
+ """Replace columns by natural key (``column_name``) — update in place
+ rather than delete-and-reinsert.
+
+ SPIKE (full-Continuum): the previous
+ delete-and-reinsert pattern produced overlapping shadow rows in
+ ``table_columns_version`` (the same ``column_name`` had a DELETE
+ shadow at tx N alongside an INSERT shadow at tx N for a fresh PK).
+ Continuum's ``Reverter`` couldn't unwind this on restore: its flush
+ ordering inserts the historical row before deleting the live one,
+ hitting the ``UNIQUE (table_id, column_name)`` constraint mid-flush
+ (ADR-004 Failure 1).
+
+ The natural-key upsert keeps PKs stable across metadata refresh.
+ Continuum captures only real field changes; new columns get plain
+ INSERT shadows; removed columns get plain DELETE shadows. No
+ natural-key collisions, so Reverter can restore cleanly.
+
+ Behaviour change vs. the previous implementation: PKs of unchanged
+ columns are preserved. Charts that reference columns by their
+ ``id`` continue to work across a metadata refresh — previously
+ such references would be invalidated.
+ """
+ existing_by_name = {c.column_name: c for c in model.columns}
+ incoming_by_name = {p["column_name"]: p for p in property_columns}
+
+ # Identity is the natural key here, never the payload's ``id``:
+ # setattr-ing an incoming ``id`` onto a name-matched row would
+ # rewrite a live primary key, and a renamed column whose payload
+ # still carries its old ``id`` would INSERT with a live PK while
+ # the old-named row is deleted in the same flush — INSERTs flush
+ # before DELETEs, so that collides on the PK / UNIQUE(table_id,
+ # column_name) constraints. ``table_id`` is pinned to *model*.
+ protected_keys = ("id", "table_id")
+
+ # Update columns present in both: in-place setattr.
+ for name, col in existing_by_name.items():
+ if name in incoming_by_name:
+ for key, value in incoming_by_name[name].items():
+ if key not in protected_keys:
+ setattr(col, key, value)
+
+ # Insert columns present only in incoming.
+ for name, properties in incoming_by_name.items():
+ if name not in existing_by_name:
+ cleaned = {
+ key: value
+ for key, value in properties.items()
+ if key not in protected_keys
+ }
+ db.session.add(TableColumn(**{**cleaned, "table_id": model.id}))
+
+ # Delete columns present only in existing.
+ for name, col in existing_by_name.items():
+ if name not in incoming_by_name:
+ db.session.delete(col)
+
+ @classmethod
+ def _upsert_columns(
+ cls, model: SqlaTable, property_columns: list[dict[str, Any]]
+ ) -> None:
+ columns_by_id = {column.id: column for column in model.columns}
Review Comment:
**Suggestion:** Add a concrete type annotation for this new ID-indexed map variable. [custom_rule]
**Severity Level:** Minor ⚠️
<details>
<summary><b>Why it matters? 🤔 </b></summary>
This is a newly introduced dictionary used as an ID lookup map and it has no explicit type annotation, so it is a real instance of the type-hint omission rule.
</details>
[](https://app.codeant.ai/fix-in-ide?tool=cursor&prompt_id=8070461118f0465a8e816d24259443c4&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset) [](https://app.codeant.ai/fix-in-ide?tool=vscode-claude&prompt_id=8070461118f0465a8e816d24259443c4&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset)
*(Use Cmd/Ctrl + Click for best experience)*
<details>
<summary><b>Prompt for AI Agent 🤖 </b></summary>
```mdx
This is a comment left during a code review.
**Path:** superset/daos/dataset.py
**Line:** 354:354
**Comment:**
*Custom Rule: Add a concrete type annotation for this new ID-indexed map variable.
Validate the correctness of the flagged issue. If correct, How can I resolve this? If you propose a fix, implement it and please make it concise.
Once fix is implemented, also check other comments on the same PR, and ask user if the user wants to fix the rest of the comments as well. if said yes, then fetch all the comments validate the correctness and implement a minimal fix
```
</details>
<a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F41076&comment_hash=9183f78847fed27264623183130f4276a8294b39ae3bea9e7ccaf7efdb92a576&reaction=like'>👍</a> | <a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F41076&comment_hash=9183f78847fed27264623183130f4276a8294b39ae3bea9e7ccaf7efdb92a576&reaction=dislike'>👎</a>
##########
superset/daos/dataset.py:
##########
@@ -363,6 +411,9 @@ def update_metrics(
- If a metric Dict does not have an `id` then we create a new metric.
- If there are extra metrics on the metadata db that are not defined on the List
then we delete.
+
+ Uses individual ORM operations (not bulk) so that SQLAlchemy-Continuum
+ can capture each row change in the version history.
"""
metrics_by_id = {metric.id: metric for metric in model.metrics}
Review Comment:
**Suggestion:** Add an explicit type annotation for this newly added metric lookup map. [custom_rule]
**Severity Level:** Minor ⚠️
<details>
<summary><b>Why it matters? 🤔 </b></summary>
This is a newly added lookup dictionary in modified code with no explicit type hint, so it violates the type-hint rule for relevant variables.
</details>
[](https://app.codeant.ai/fix-in-ide?tool=cursor&prompt_id=632e957547c1442885dd3172c4693441&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset) [](https://app.codeant.ai/fix-in-ide?tool=vscode-claude&prompt_id=632e957547c1442885dd3172c4693441&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset)
*(Use Cmd/Ctrl + Click for best experience)*
<details>
<summary><b>Prompt for AI Agent 🤖 </b></summary>
```mdx
This is a comment left during a code review.
**Path:** superset/daos/dataset.py
**Line:** 419:419
**Comment:**
*Custom Rule: Add an explicit type annotation for this newly added metric lookup map.
Validate the correctness of the flagged issue. If correct, How can I resolve this? If you propose a fix, implement it and please make it concise.
Once fix is implemented, also check other comments on the same PR, and ask user if the user wants to fix the rest of the comments as well. if said yes, then fetch all the comments validate the correctness and implement a minimal fix
```
</details>
<a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F41076&comment_hash=2a391dcab0b9b7406143a9d68ba13573466f7c6f2d1dd8600d5fef1cc8e17bbd&reaction=like'>👍</a> | <a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F41076&comment_hash=2a391dcab0b9b7406143a9d68ba13573466f7c6f2d1dd8600d5fef1cc8e17bbd&reaction=dislike'>👎</a>
##########
superset/daos/version.py:
##########
@@ -0,0 +1,69 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Backward-compat façade for the entity-versioning DAO surface.
+
+The actual implementation lives in :mod:`superset.versioning.queries`
+(read side: list/get/resolve/find/UUID derivation). This module
+re-exports it under a single ``VersionDAO`` class plus the module-level
+UUID helpers so existing callers keep working without changes. (The
+write side — restore + audit stamping — ships in a later PR; only the
+read surface is wired here.)
+
+New code should import from the versioning sub-modules directly.
+"""
+
+from __future__ import annotations
+
+from superset.versioning.queries import (
+ current_live_transaction_id,
+ current_live_version_uuid,
+ current_version_number,
+ derive_version_uuid,
+ derive_version_uuid as _derive_version_uuid, # noqa: F401
+ find_active_by_uuid,
+ get_version,
+ list_change_records_batch,
+ list_versions,
+ resolve_version_uuid,
+ VERSION_UUID_NAMESPACE,
+)
+
+# Re-exports for ``from superset.daos.version import …`` consumers.
+__all__ = [
+ "VERSION_UUID_NAMESPACE",
+ "VersionDAO",
+ "derive_version_uuid",
+]
Review Comment:
**Suggestion:** Add an explicit type annotation for the exported symbol list to satisfy the type-hinting requirement for annotatable module-level variables. [custom_rule]
**Severity Level:** Minor ⚠️
<details>
<summary><b>Why it matters? 🤔 </b></summary>
The module-level exported symbol list is a clearly annotatable variable, and the file does not provide a type hint for it. This matches the rule requiring type hints on relevant variables that can be annotated.
</details>
[](https://app.codeant.ai/fix-in-ide?tool=cursor&prompt_id=e486b0d74eb54720b9bb79a18272b144&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset) [](https://app.codeant.ai/fix-in-ide?tool=vscode-claude&prompt_id=e486b0d74eb54720b9bb79a18272b144&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset)
*(Use Cmd/Ctrl + Click for best experience)*
<details>
<summary><b>Prompt for AI Agent 🤖 </b></summary>
```mdx
This is a comment left during a code review.
**Path:** superset/daos/version.py
**Line:** 46:50
**Comment:**
*Custom Rule: Add an explicit type annotation for the exported symbol list to satisfy the type-hinting requirement for annotatable module-level variables.
Validate the correctness of the flagged issue. If correct, How can I resolve this? If you propose a fix, implement it and please make it concise.
Once fix is implemented, also check other comments on the same PR, and ask user if the user wants to fix the rest of the comments as well. if said yes, then fetch all the comments validate the correctness and implement a minimal fix
```
</details>
<a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F41076&comment_hash=a6040fea15580f3f8b72c657809aa393a3331e97c80250e7a1dc1e32a77d8c0c&reaction=like'>👍</a> | <a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F41076&comment_hash=a6040fea15580f3f8b72c657809aa393a3331e97c80250e7a1dc1e32a77d8c0c&reaction=dislike'>👎</a>
##########
superset/datasets/api.py:
##########
@@ -440,17 +504,69 @@ def put(self, pk: int) -> Response:
# This validates custom Schema with custom validations
except ValidationError as error:
return self.response_400(message=error.messages)
+
+ # Live version identifiers before the update (empty + query-free when
+ # ``ENABLE_VERSIONING_CAPTURE`` is off).
+ old_info = current_entity_version_info(SqlaTable, pk)
Review Comment:
**Suggestion:** Add an explicit type annotation to this newly introduced variable assignment so the version metadata object is typed. [custom_rule]
**Severity Level:** Minor ⚠️
<details>
<summary><b>Why it matters? 🤔 </b></summary>
The new local variable is introduced without an explicit type annotation, and it is a relevant variable that can be annotated under the Python type-hint rule.
</details>
[](https://app.codeant.ai/fix-in-ide?tool=cursor&prompt_id=87f987b34e92458195f1b7d6be3b7154&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset) [](https://app.codeant.ai/fix-in-ide?tool=vscode-claude&prompt_id=87f987b34e92458195f1b7d6be3b7154&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset)
*(Use Cmd/Ctrl + Click for best experience)*
<details>
<summary><b>Prompt for AI Agent 🤖 </b></summary>
```mdx
This is a comment left during a code review.
**Path:** superset/datasets/api.py
**Line:** 510:510
**Comment:**
*Custom Rule: Add an explicit type annotation to this newly introduced variable assignment so the version metadata object is typed.
Validate the correctness of the flagged issue. If correct, How can I resolve this? If you propose a fix, implement it and please make it concise.
Once fix is implemented, also check other comments on the same PR, and ask user if the user wants to fix the rest of the comments as well. if said yes, then fetch all the comments validate the correctness and implement a minimal fix
```
</details>
<a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F41076&comment_hash=829d471f5b448dd875154ca0a3910293abab2e3c814e8fda1762852e12072399&reaction=like'>👍</a> | <a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F41076&comment_hash=829d471f5b448dd875154ca0a3910293abab2e3c814e8fda1762852e12072399&reaction=dislike'>👎</a>
##########
superset/datasets/api.py:
##########
@@ -440,17 +504,69 @@ def put(self, pk: int) -> Response:
# This validates custom Schema with custom validations
except ValidationError as error:
return self.response_400(message=error.messages)
+
+ # Live version identifiers before the update (empty + query-free when
+ # ``ENABLE_VERSIONING_CAPTURE`` is off).
+ old_info = current_entity_version_info(SqlaTable, pk)
+
try:
+ # Two commands, two commits, two Continuum transactions for an
+ # ``override_columns`` save — deliberately NOT merged into one
+ # transaction. A single-transaction design was attempted and
+ # reverted: ``DBEventLogger`` writes request logs through the
+ # SHARED scoped session and calls ``commit()`` /
+ # ``rollback()`` on it mid-request (superset/utils/log.py),
+ # so any save held uncommitted across a logged sub-action can
+ # be committed half-done (Postgres/MySQL) or rolled back
+ # entirely on a transient logger failure (SQLite's
+ # "database is locked"). Until the event logger gets its own
+ # session, per-command commit boundaries are the only shape
+ # whose failure modes are honest. Consequence the
+ # version-history UI must tolerate: one logical save can
+ # surface as two version transactions stamped the same second.
changed_model = UpdateDatasetCommand(pk, item, override_columns).run()
+ # Capture the post-update identifiers BEFORE the refresh:
+ # RefreshDatasetCommand commits its own transaction, so reading
+ # afterwards would attribute the refresh's version to the
+ # user's update (and old→new would span two transactions).
+ new_info = current_entity_version_info(
+ SqlaTable, changed_model.id, changed_model.uuid
+ )
+ etag_version_uuid = new_info.version_uuid
Review Comment:
**Suggestion:** Annotate this new variable with its concrete optional string type to comply with the type-hint requirement for relevant variables. [custom_rule]
**Severity Level:** Minor ⚠️
<details>
<summary><b>Why it matters? 🤔 </b></summary>
The variable is introduced without a type annotation and is a simple local value that can be annotated, so this is a real omission under the type-hint rule.
</details>
[](https://app.codeant.ai/fix-in-ide?tool=cursor&prompt_id=ad94bb4d9008401d878241e8aacfcb6d&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset) [](https://app.codeant.ai/fix-in-ide?tool=vscode-claude&prompt_id=ad94bb4d9008401d878241e8aacfcb6d&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset)
*(Use Cmd/Ctrl + Click for best experience)*
<details>
<summary><b>Prompt for AI Agent 🤖 </b></summary>
```mdx
This is a comment left during a code review.
**Path:** superset/datasets/api.py
**Line:** 535:535
**Comment:**
*Custom Rule: Annotate this new variable with its concrete optional string type to comply with the type-hint requirement for relevant variables.
Validate the correctness of the flagged issue. If correct, How can I resolve this? If you propose a fix, implement it and please make it concise.
Once fix is implemented, also check other comments on the same PR, and ask user if the user wants to fix the rest of the comments as well. if said yes, then fetch all the comments validate the correctness and implement a minimal fix
```
</details>
<a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F41076&comment_hash=73b727b08894f5c1a9906245c3dfcfeec8bdcd8485625070e53a39168ce6a881&reaction=like'>👍</a> | <a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F41076&comment_hash=73b727b08894f5c1a9906245c3dfcfeec8bdcd8485625070e53a39168ce6a881&reaction=dislike'>👎</a>
##########
superset/datasets/api.py:
##########
@@ -440,17 +504,69 @@ def put(self, pk: int) -> Response:
# This validates custom Schema with custom validations
except ValidationError as error:
return self.response_400(message=error.messages)
+
+ # Live version identifiers before the update (empty + query-free when
+ # ``ENABLE_VERSIONING_CAPTURE`` is off).
+ old_info = current_entity_version_info(SqlaTable, pk)
+
try:
+ # Two commands, two commits, two Continuum transactions for an
+ # ``override_columns`` save — deliberately NOT merged into one
+ # transaction. A single-transaction design was attempted and
+ # reverted: ``DBEventLogger`` writes request logs through the
+ # SHARED scoped session and calls ``commit()`` /
+ # ``rollback()`` on it mid-request (superset/utils/log.py),
+ # so any save held uncommitted across a logged sub-action can
+ # be committed half-done (Postgres/MySQL) or rolled back
+ # entirely on a transient logger failure (SQLite's
+ # "database is locked"). Until the event logger gets its own
+ # session, per-command commit boundaries are the only shape
+ # whose failure modes are honest. Consequence the
+ # version-history UI must tolerate: one logical save can
+ # surface as two version transactions stamped the same second.
changed_model = UpdateDatasetCommand(pk, item, override_columns).run()
+ # Capture the post-update identifiers BEFORE the refresh:
+ # RefreshDatasetCommand commits its own transaction, so reading
+ # afterwards would attribute the refresh's version to the
+ # user's update (and old→new would span two transactions).
+ new_info = current_entity_version_info(
+ SqlaTable, changed_model.id, changed_model.uuid
+ )
Review Comment:
**Suggestion:** Add an explicit type annotation for this new local variable returned from the version helper to satisfy required type hint coverage. [custom_rule]
**Severity Level:** Minor ⚠️
<details>
<summary><b>Why it matters? 🤔 </b></summary>
This newly added local variable is also unannotated even though its type is inferable, so it matches the rule requiring type hints for relevant variables.
</details>
[](https://app.codeant.ai/fix-in-ide?tool=cursor&prompt_id=62f18726ed7d449e8bbfb692915da5ad&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset) [](https://app.codeant.ai/fix-in-ide?tool=vscode-claude&prompt_id=62f18726ed7d449e8bbfb692915da5ad&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset)
*(Use Cmd/Ctrl + Click for best experience)*
<details>
<summary><b>Prompt for AI Agent 🤖 </b></summary>
```mdx
This is a comment left during a code review.
**Path:** superset/datasets/api.py
**Line:** 532:534
**Comment:**
*Custom Rule: Add an explicit type annotation for this new local variable returned from the version helper to satisfy required type hint coverage.
Validate the correctness of the flagged issue. If correct, How can I resolve this? If you propose a fix, implement it and please make it concise.
Once fix is implemented, also check other comments on the same PR, and ask user if the user wants to fix the rest of the comments as well. if said yes, then fetch all the comments validate the correctness and implement a minimal fix
```
</details>
<a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F41076&comment_hash=fafc9de81b73e957b35bde969ac793e82a14637810636f0ad6abf1e4a0ed82de&reaction=like'>👍</a> | <a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F41076&comment_hash=fafc9de81b73e957b35bde969ac793e82a14637810636f0ad6abf1e4a0ed82de&reaction=dislike'>👎</a>
##########
scripts/seed_junction_load.py:
##########
@@ -0,0 +1,679 @@
+#!/usr/bin/env python3
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+# ----------------------------------------------------------------------
+# Stress-test data generator for the composite-PK migration (sc-105349).
+#
+# Bulk-inserts synthetic parent rows and many-to-many junction rows for
+# the eight association tables that the composite-PK migration touches.
+# Useful for measuring migration runtime at varying scales — run this at
+# 100K / 1M / 5M / 10M rows and time the migration at each scale to
+# verify the O(N log N) extrapolation.
+#
+# Idempotent: rerunning with the same target is a no-op; rerunning with
+# a higher target adds rows up to the new total. Batched bulk INSERTs
+# (10K rows per statement) make it fast on Postgres, MySQL, and SQLite.
+#
+# Usage (inside the Superset container):
+#
+# docker exec superset-superset-1 \\
+# /app/.venv/bin/python /app/scripts/seed_junction_load.py \\
+# --dashboard-slices 1000000 \\
+# --slice-user 100000 \\
+# --dashboard-user 100000
+#
+# Run with no flags for the defaults shown below. Use ``--dry-run`` to
+# print the planned inserts without writing anything.
+#
+# The script connects via Superset's standard ``DATABASE_*`` env vars
+# (or ``SUPERSET__SQLALCHEMY_DATABASE_URI`` if set), so it works
+# automatically inside the Superset container regardless of which
+# metadata DB backend is in use.
+
+from __future__ import annotations
+
+import argparse
+import logging
+import os
+import sys
+import time
+from contextlib import contextmanager
+from typing import Iterator
+from uuid import uuid4
+
+import sqlalchemy as sa
+from sqlalchemy.engine import Connection, Engine
+
+logger = logging.getLogger("seed_junction_load")
Review Comment:
**Suggestion:** Add an explicit type annotation to this module-level logger variable (for example, annotate it as a logging logger type) to satisfy the type-hint requirement for relevant variables. [custom_rule]
**Severity Level:** Minor ⚠️
<details>
<summary><b>Why it matters? 🤔 </b></summary>
The module defines a reusable logger without an explicit type annotation, which matches the custom rule requiring type hints on relevant variables that can be annotated.
</details>
[](https://app.codeant.ai/fix-in-ide?tool=cursor&prompt_id=5c77e170ef5145c2856ba48ad38cdd09&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset) [](https://app.codeant.ai/fix-in-ide?tool=vscode-claude&prompt_id=5c77e170ef5145c2856ba48ad38cdd09&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset)
*(Use Cmd/Ctrl + Click for best experience)*
<details>
<summary><b>Prompt for AI Agent 🤖 </b></summary>
```mdx
This is a comment left during a code review.
**Path:** scripts/seed_junction_load.py
**Line:** 62:62
**Comment:**
*Custom Rule: Add an explicit type annotation to this module-level logger variable (for example, annotate it as a logging logger type) to satisfy the type-hint requirement for relevant variables.
Validate the correctness of the flagged issue. If correct, How can I resolve this? If you propose a fix, implement it and please make it concise.
Once fix is implemented, also check other comments on the same PR, and ask user if the user wants to fix the rest of the comments as well. if said yes, then fetch all the comments validate the correctness and implement a minimal fix
```
</details>
<a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F41076&comment_hash=31d0563a060dc4ed5b4688fe97728c1db4d132102db5695689e3871ae38ceb80&reaction=like'>👍</a> | <a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F41076&comment_hash=31d0563a060dc4ed5b4688fe97728c1db4d132102db5695689e3871ae38ceb80&reaction=dislike'>👎</a>
##########
superset/charts/api.py:
##########
@@ -437,9 +496,29 @@ def put(self, pk: int) -> Response:
# This validates custom Schema with custom validations
except ValidationError as error:
return self.response_400(message=error.messages)
+
+ # Live version identifiers before the update (empty + query-free when
+ # ``ENABLE_VERSIONING_CAPTURE`` is off, so this stays inert under the
+ # kill-switch).
+ old_info = current_entity_version_info(Slice, pk)
Review Comment:
**Suggestion:** Add explicit type annotations for the new version-info local variables to satisfy the type-hint requirement for newly introduced variables. [custom_rule]
**Severity Level:** Minor ⚠️
<details>
<summary><b>Why it matters? 🤔 </b></summary>
This newly added local variable is not type-annotated, and it is a value whose type can reasonably be annotated under the Python type-hint rule for relevant variables.
</details>
[](https://app.codeant.ai/fix-in-ide?tool=cursor&prompt_id=ff9dd4f2faa74127bd3c7f1e787feba6&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset) [](https://app.codeant.ai/fix-in-ide?tool=vscode-claude&prompt_id=ff9dd4f2faa74127bd3c7f1e787feba6&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset)
*(Use Cmd/Ctrl + Click for best experience)*
<details>
<summary><b>Prompt for AI Agent 🤖 </b></summary>
```mdx
This is a comment left during a code review.
**Path:** superset/charts/api.py
**Line:** 503:503
**Comment:**
*Custom Rule: Add explicit type annotations for the new version-info local variables to satisfy the type-hint requirement for newly introduced variables.
Validate the correctness of the flagged issue. If correct, How can I resolve this? If you propose a fix, implement it and please make it concise.
Once fix is implemented, also check other comments on the same PR, and ask user if the user wants to fix the rest of the comments as well. if said yes, then fetch all the comments validate the correctness and implement a minimal fix
```
</details>
<a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F41076&comment_hash=37b39fba1a3763e0b200e1ba3ecdb300a9be6e8cba956fab4ed5680638a495c7&reaction=like'>👍</a> | <a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F41076&comment_hash=37b39fba1a3763e0b200e1ba3ecdb300a9be6e8cba956fab4ed5680638a495c7&reaction=dislike'>👎</a>
##########
scripts/seed_junction_load.py:
##########
@@ -0,0 +1,679 @@
+#!/usr/bin/env python3
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+# ----------------------------------------------------------------------
+# Stress-test data generator for the composite-PK migration (sc-105349).
+#
+# Bulk-inserts synthetic parent rows and many-to-many junction rows for
+# the eight association tables that the composite-PK migration touches.
+# Useful for measuring migration runtime at varying scales — run this at
+# 100K / 1M / 5M / 10M rows and time the migration at each scale to
+# verify the O(N log N) extrapolation.
+#
+# Idempotent: rerunning with the same target is a no-op; rerunning with
+# a higher target adds rows up to the new total. Batched bulk INSERTs
+# (10K rows per statement) make it fast on Postgres, MySQL, and SQLite.
+#
+# Usage (inside the Superset container):
+#
+# docker exec superset-superset-1 \\
+# /app/.venv/bin/python /app/scripts/seed_junction_load.py \\
+# --dashboard-slices 1000000 \\
+# --slice-user 100000 \\
+# --dashboard-user 100000
+#
+# Run with no flags for the defaults shown below. Use ``--dry-run`` to
+# print the planned inserts without writing anything.
+#
+# The script connects via Superset's standard ``DATABASE_*`` env vars
+# (or ``SUPERSET__SQLALCHEMY_DATABASE_URI`` if set), so it works
+# automatically inside the Superset container regardless of which
+# metadata DB backend is in use.
+
+from __future__ import annotations
+
+import argparse
+import logging
+import os
+import sys
+import time
+from contextlib import contextmanager
+from typing import Iterator
+from uuid import uuid4
+
+import sqlalchemy as sa
+from sqlalchemy.engine import Connection, Engine
+
+logger = logging.getLogger("seed_junction_load")
+
+# Bulk INSERT batch size. Larger values = fewer statements but more memory.
+BATCH = 10_000
Review Comment:
**Suggestion:** Annotate this module-level constant with an explicit integer type so the variable is fully type-hinted. [custom_rule]
**Severity Level:** Minor ⚠️
<details>
<summary><b>Why it matters? 🤔 </b></summary>
This module-level integer constant is unannotated even though it can easily be typed, so it fits the Python type-hint rule for relevant variables.
</details>
[](https://app.codeant.ai/fix-in-ide?tool=cursor&prompt_id=c51526c5a0164da6bbbdcfa00c68778b&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset) [](https://app.codeant.ai/fix-in-ide?tool=vscode-claude&prompt_id=c51526c5a0164da6bbbdcfa00c68778b&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset)
*(Use Cmd/Ctrl + Click for best experience)*
<details>
<summary><b>Prompt for AI Agent 🤖 </b></summary>
```mdx
This is a comment left during a code review.
**Path:** scripts/seed_junction_load.py
**Line:** 65:65
**Comment:**
*Custom Rule: Annotate this module-level constant with an explicit integer type so the variable is fully type-hinted.
Validate the correctness of the flagged issue. If correct, How can I resolve this? If you propose a fix, implement it and please make it concise.
Once fix is implemented, also check other comments on the same PR, and ask user if the user wants to fix the rest of the comments as well. if said yes, then fetch all the comments validate the correctness and implement a minimal fix
```
</details>
<a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F41076&comment_hash=faa76e5edb16513ebaa7750956d7e861fb3935e1db206e4bd31bff561bf96524&reaction=like'>👍</a> | <a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F41076&comment_hash=faa76e5edb16513ebaa7750956d7e861fb3935e1db206e4bd31bff561bf96524&reaction=dislike'>👎</a>
##########
scripts/seed_junction_load.py:
##########
@@ -0,0 +1,679 @@
+#!/usr/bin/env python3
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+# ----------------------------------------------------------------------
+# Stress-test data generator for the composite-PK migration (sc-105349).
+#
+# Bulk-inserts synthetic parent rows and many-to-many junction rows for
+# the eight association tables that the composite-PK migration touches.
+# Useful for measuring migration runtime at varying scales — run this at
+# 100K / 1M / 5M / 10M rows and time the migration at each scale to
+# verify the O(N log N) extrapolation.
+#
+# Idempotent: rerunning with the same target is a no-op; rerunning with
+# a higher target adds rows up to the new total. Batched bulk INSERTs
+# (10K rows per statement) make it fast on Postgres, MySQL, and SQLite.
+#
+# Usage (inside the Superset container):
+#
+# docker exec superset-superset-1 \\
+# /app/.venv/bin/python /app/scripts/seed_junction_load.py \\
+# --dashboard-slices 1000000 \\
+# --slice-user 100000 \\
+# --dashboard-user 100000
+#
+# Run with no flags for the defaults shown below. Use ``--dry-run`` to
+# print the planned inserts without writing anything.
+#
+# The script connects via Superset's standard ``DATABASE_*`` env vars
+# (or ``SUPERSET__SQLALCHEMY_DATABASE_URI`` if set), so it works
+# automatically inside the Superset container regardless of which
+# metadata DB backend is in use.
+
+from __future__ import annotations
+
+import argparse
+import logging
+import os
+import sys
+import time
+from contextlib import contextmanager
+from typing import Iterator
+from uuid import uuid4
+
+import sqlalchemy as sa
+from sqlalchemy.engine import Connection, Engine
+
+logger = logging.getLogger("seed_junction_load")
+
+# Bulk INSERT batch size. Larger values = fewer statements but more memory.
+BATCH = 10_000
+
+# Default per-junction-table target row counts. Tuned to mimic the shape
+# of a large multi-team Superset install. Override via CLI flags.
+DEFAULTS: dict[str, int] = {
+ "dashboard_slices": 1_000_000,
+ "slice_user": 100_000,
+ "dashboard_user": 100_000,
+ "dashboard_roles": 10_000,
+}
+
+# (junction_table, fk1_col, fk2_col, parent1_table, parent2_table)
+# parents reference id columns; we generate (fk1, fk2) pairs by sampling
+# from the parents' existing IDs.
+JUNCTIONS: list[tuple[str, str, str, str, str]] = [
+ ("dashboard_slices", "dashboard_id", "slice_id", "dashboards", "slices"),
+ ("slice_user", "user_id", "slice_id", "ab_user", "slices"),
+ ("dashboard_user", "user_id", "dashboard_id", "ab_user", "dashboards"),
+ ("dashboard_roles", "dashboard_id", "role_id", "dashboards", "ab_role"),
+]
+
+# Junction tables that originally carried ``UNIQUE(fk1, fk2)`` and therefore
+# cannot accept duplicate ``(fk1, fk2)`` pairs even on the pre-migration
+# (downgrade) schema. The other JUNCTIONS allow duplicates pre-migration.
+# Only ``dashboard_slices`` is listed: the migration's other UNIQUE table
+# (``report_schedule_user``) is not in JUNCTIONS — this script doesn't seed
+# it — so listing it here would imply coverage that doesn't exist. Add it
+# alongside a JUNCTIONS entry if that table ever gets seeded.
+JUNCTIONS_WITH_UNIQUE: set[str] = {"dashboard_slices"}
+
+
+# ----------------------------------------------------------------------
+# Connection setup
+# ----------------------------------------------------------------------
+
+
+def build_engine() -> Engine:
+ """Build a SQLAlchemy engine from Superset env vars."""
+ if uri := os.environ.get("SUPERSET__SQLALCHEMY_DATABASE_URI"):
+ logger.info("Using SUPERSET__SQLALCHEMY_DATABASE_URI from env")
+ return sa.create_engine(uri)
+
+ try:
+ dialect = os.environ["DATABASE_DIALECT"]
+ user = os.environ["DATABASE_USER"]
+ password = os.environ["DATABASE_PASSWORD"]
+ host = os.environ["DATABASE_HOST"]
+ port = os.environ["DATABASE_PORT"]
+ db = os.environ["DATABASE_DB"]
+ except KeyError as exc:
+ sys.exit(
+ f"Missing env var {exc}; either set DATABASE_DIALECT/USER/PASSWORD/"
+ f"HOST/PORT/DB or SUPERSET__SQLALCHEMY_DATABASE_URI before running."
+ )
+
+ uri = f"{dialect}://{user}:{password}@{host}:{port}/{db}"
+ logger.info(
+ "Built URI from DATABASE_* env vars (dialect=%s, host=%s)", dialect, host
+ )
+ return sa.create_engine(uri)
+
+
+# ----------------------------------------------------------------------
+# Helpers
+# ----------------------------------------------------------------------
+
+
+def uuid_value(dialect_name: str) -> bytes | str:
+ """Return a UUID in the form the active dialect expects.
+
+ MySQL stores UUIDs as ``BINARY(16)`` (16 raw bytes); Postgres has a
+ native ``UUID`` type that accepts strings; SQLite stores them as
+ BLOB/TEXT and accepts either. Branching here keeps the seed script
+ backend-agnostic without depending on Superset's custom column types.
+ """
+ if dialect_name.startswith("mysql"):
+ return uuid4().bytes
+ return str(uuid4())
+
+
+@contextmanager
+def time_phase(name: str) -> Iterator[None]:
+ """Log elapsed wall time for a named phase."""
+ start = time.monotonic()
+ logger.info("[%s] starting", name)
+ try:
+ yield
+ finally:
+ elapsed = time.monotonic() - start
+ logger.info("[%s] done in %.2fs", name, elapsed)
+
+
+def count_rows(conn: Connection, table: str) -> int:
+ return conn.scalar(sa.text(f"SELECT COUNT(*) FROM {table}")) or 0 # noqa: S608
+
+
+def existing_ids(conn: Connection, table: str, limit: int | None = None) -> list[int]:
+ sql = f"SELECT id FROM {table} ORDER BY id" # noqa: S608
+ if limit is not None:
+ sql += f" LIMIT {limit}"
+ return [row[0] for row in conn.execute(sa.text(sql))]
+
+
+# ----------------------------------------------------------------------
+# Parent seeders
+#
+# Each function ensures the named parent table has at least ``target``
+# rows by inserting synthetic ones with minimal-but-valid columns.
+# Returns nothing; subsequent code reads back IDs via ``existing_ids``.
+# ----------------------------------------------------------------------
+
+
+def seed_dashboards(conn: Connection, target: int, dry_run: bool) -> None:
+ current = count_rows(conn, "dashboards")
+ if current >= target:
+ logger.info(
+ "dashboards: %d rows (target %d) — no insert needed", current, target
+ )
+ return
+ needed = target - current
+ logger.info("dashboards: %d → %d (+%d)", current, target, needed)
+ if dry_run:
+ return
+
+ dialect = conn.engine.dialect.name
+ sql = sa.text(
+ "INSERT INTO dashboards (uuid, dashboard_title, slug, published) "
+ "VALUES (:uuid, :title, :slug, :published)"
+ )
+ for batch_start in range(0, needed, BATCH):
+ rows = [
+ {
+ "uuid": uuid_value(dialect),
+ "title": f"seed_dashboard_{current + i}",
+ "slug": f"seed-dashboard-{current + i}-{uuid4().hex[:8]}",
+ "published": False,
+ }
+ for i in range(batch_start, min(batch_start + BATCH, needed))
+ ]
+ conn.execute(sql, rows)
+ logger.info(" dashboards: inserted %d / %d", batch_start + len(rows), needed)
+
+
+def seed_dbs(conn: Connection, dry_run: bool) -> int:
+ """Ensure at least one row exists in ``dbs`` (parent of ``tables``).
+ Returns the id to use as ``database_id`` when seeding ``tables``."""
+ ids = existing_ids(conn, "dbs", limit=1)
+ if ids:
+ return ids[0]
+ if dry_run:
+ return -1 # placeholder
+ dialect = conn.engine.dialect.name
+ logger.info("dbs: inserting one synthetic database (no rows present)")
+ conn.execute(
+ sa.text(
+ "INSERT INTO dbs (uuid, database_name, sqlalchemy_uri, expose_in_sqllab) "
+ "VALUES (:uuid, :name, :uri, :expose)"
+ ),
+ {
+ "uuid": uuid_value(dialect),
+ "name": f"seed_db_{uuid4().hex[:8]}",
+ "uri": "sqlite:///seed.db",
+ "expose": False,
+ },
+ )
+ return existing_ids(conn, "dbs", limit=1)[0]
+
+
+def seed_tables(conn: Connection, target: int, dry_run: bool) -> None:
+ current = count_rows(conn, "tables")
+ if current >= target:
+ logger.info("tables: %d rows (target %d) — no insert needed", current, target)
+ return
+ needed = target - current
+ logger.info("tables: %d → %d (+%d)", current, target, needed)
+ if dry_run:
+ return
+
+ database_id = seed_dbs(conn, dry_run=False)
+ dialect = conn.engine.dialect.name
+ sql = sa.text(
+ "INSERT INTO tables (uuid, table_name, database_id) "
+ "VALUES (:uuid, :name, :db_id)"
+ )
+ for batch_start in range(0, needed, BATCH):
+ rows = [
+ {
+ "uuid": uuid_value(dialect),
+ "name": f"seed_table_{current + i}",
+ "db_id": database_id,
+ }
+ for i in range(batch_start, min(batch_start + BATCH, needed))
+ ]
+ conn.execute(sql, rows)
+ logger.info(" tables: inserted %d / %d", batch_start + len(rows), needed)
+
+
+def seed_slices(conn: Connection, target: int, dry_run: bool) -> None:
+ current = count_rows(conn, "slices")
+ if current >= target:
+ logger.info("slices: %d rows (target %d) — no insert needed", current, target)
+ return
+ needed = target - current
+ logger.info("slices: %d → %d (+%d)", current, target, needed)
+ if dry_run:
+ return
+
+ # Slices reference tables.id; ensure at least one ``tables`` row exists
+ # so the FK is satisfiable (datasource_id is nullable but we set it for
+ # realism). The migration test doesn't care, but a real Superset that
+ # re-renders these slices does.
+ seed_tables(conn, target=1, dry_run=False)
+ table_id = existing_ids(conn, "tables", limit=1)[0]
+ dialect = conn.engine.dialect.name
+ sql = sa.text(
+ "INSERT INTO slices "
+ "(uuid, slice_name, datasource_id, datasource_type, viz_type) "
+ "VALUES (:uuid, :name, :ds_id, :ds_type, :viz)"
+ )
+ for batch_start in range(0, needed, BATCH):
+ rows = [
+ {
+ "uuid": uuid_value(dialect),
+ "name": f"seed_slice_{current + i}",
+ "ds_id": table_id,
+ "ds_type": "table",
+ "viz": "table",
+ }
+ for i in range(batch_start, min(batch_start + BATCH, needed))
+ ]
+ conn.execute(sql, rows)
+ logger.info(" slices: inserted %d / %d", batch_start + len(rows), needed)
+
+
+def seed_users(conn: Connection, target: int, dry_run: bool) -> None:
+ current = count_rows(conn, "ab_user")
+ if current >= target:
+ logger.info("ab_user: %d rows (target %d) — no insert needed", current, target)
+ return
+ needed = target - current
+ logger.info("ab_user: %d → %d (+%d)", current, target, needed)
+ if dry_run:
+ return
+
+ sql = sa.text(
+ "INSERT INTO ab_user (first_name, last_name, username, email, active) "
+ "VALUES (:first, :last, :username, :email, :active)"
+ )
+ for batch_start in range(0, needed, BATCH):
+ rows = [
+ {
+ "first": "seed",
+ "last": f"user_{current + i}",
+ "username": f"seed_user_{current + i}_{uuid4().hex[:8]}",
+ "email": f"seed_user_{current + i}_{uuid4().hex[:8]}@example.invalid",
+ "active": True,
+ }
+ for i in range(batch_start, min(batch_start + BATCH, needed))
+ ]
+ conn.execute(sql, rows)
+ logger.info(" ab_user: inserted %d / %d", batch_start + len(rows), needed)
+
+
+def seed_roles(conn: Connection, target: int, dry_run: bool) -> None:
+ current = count_rows(conn, "ab_role")
+ if current >= target:
+ logger.info("ab_role: %d rows (target %d) — no insert needed", current, target)
+ return
+ needed = target - current
+ logger.info("ab_role: %d → %d (+%d)", current, target, needed)
+ if dry_run:
+ return
+
+ sql = sa.text("INSERT INTO ab_role (name) VALUES (:name)")
+ for batch_start in range(0, needed, BATCH):
+ rows = [
+ {"name": f"seed_role_{current + i}_{uuid4().hex[:8]}"}
+ for i in range(batch_start, min(batch_start + BATCH, needed))
+ ]
+ conn.execute(sql, rows)
+ logger.info(" ab_role: inserted %d / %d", batch_start + len(rows), needed)
+
+
+# ----------------------------------------------------------------------
+# Junction seeder
+# ----------------------------------------------------------------------
+
+
+def _load_existing_pairs(
+ conn: Connection, junction: str, fk1_col: str, fk2_col: str
+) -> set[tuple[int, int]]:
+ """Load existing ``(fk1, fk2)`` pairs from a junction table into a set.
+
+ Used so the seeder can skip them when generating new pairs (junction
+ tables enforce uniqueness on the FK pair). Memory is ~32 bytes/tuple
+ on CPython, so 10M existing pairs is ~320MB — acceptable for a dev
+ machine. The junction / column names come from ``JUNCTIONS``, not
+ user input, so the f-string interpolation is safe.
+ """
+ sql_text = f"SELECT {fk1_col}, {fk2_col} FROM {junction}" # noqa: S608
+ return {(row[0], row[1]) for row in conn.execute(sa.text(sql_text))}
+
+
+def _generate_new_pairs(
+ p1_ids: list[int],
+ p2_ids: list[int],
+ existing_pairs: set[tuple[int, int]],
+) -> Iterator[tuple[int, int]]:
+ """Yield ``(fk1, fk2)`` pairs from the parent1 × parent2 cross-product
+ that are not already in ``existing_pairs``."""
+ for fk1 in p1_ids:
+ for fk2 in p2_ids:
+ if (fk1, fk2) not in existing_pairs:
+ yield (fk1, fk2)
+
+
+def seed_junction(
+ conn: Connection,
+ junction: str,
+ fk1_col: str,
+ fk2_col: str,
+ parent1: str,
+ parent2: str,
+ target: int,
+ dry_run: bool,
+) -> None:
+ """Bulk-insert junction rows up to ``target`` rows total.
+
+ Generates ``(fk1, fk2)`` pairs by walking the cross-product of
+ parent1 IDs × parent2 IDs in row-major order, skipping pairs that
+ already exist. Walking the cross-product deterministically keeps
+ the script replayable: re-running with the same target is a no-op,
+ and re-running with a higher target appends new pairs in a stable
+ order regardless of how many runs preceded.
+ """
+ current = count_rows(conn, junction)
+ if current >= target:
+ logger.info(
+ "%s: %d rows (target %d) — no insert needed", junction, current, target
+ )
+ return
+ needed = target - current
+ logger.info("%s: %d → %d (+%d)", junction, current, target, needed)
+ if dry_run:
+ return
+
+ p1_ids = existing_ids(conn, parent1)
+ p2_ids = existing_ids(conn, parent2)
+ max_pairs = len(p1_ids) * len(p2_ids)
+ if max_pairs < target:
+ sys.exit(
+ f"Cannot reach {target} rows in {junction}: "
+ f"only {max_pairs} unique pairs available "
+ f"({len(p1_ids)} × {len(p2_ids)}). "
+ f"Increase parent targets and rerun."
+ )
+
+ existing_pairs: set[tuple[int, int]] = (
+ _load_existing_pairs(conn, junction, fk1_col, fk2_col) if current > 0 else set()
+ )
+ if existing_pairs:
+ logger.info(
+ " %s: loaded %d existing pairs into avoidance set",
+ junction,
+ len(existing_pairs),
+ )
+
+ insert_sql = sa.text(
+ f"INSERT INTO {junction} ({fk1_col}, {fk2_col}) " # noqa: S608
+ f"VALUES (:fk1, :fk2)"
+ )
+
+ inserted = 0
+ batch: list[dict[str, int]] = []
+ for fk1, fk2 in _generate_new_pairs(p1_ids, p2_ids, existing_pairs):
+ batch.append({"fk1": fk1, "fk2": fk2})
+ inserted += 1
+ if len(batch) == BATCH or inserted == needed:
+ conn.execute(insert_sql, batch)
+ logger.info(" %s: inserted %d / %d", junction, inserted, needed)
+ batch = []
+ if inserted == needed:
+ return
+ if inserted < needed:
+ sys.exit(
+ f"Ran out of unique pairs at {inserted}/{needed} for {junction} "
+ f"(parents have {len(p1_ids)} × {len(p2_ids)} = {max_pairs} pairs, "
+ f"{len(existing_pairs)} already present)"
+ )
+
+
+# ----------------------------------------------------------------------
+# Orchestration
+# ----------------------------------------------------------------------
+
+
+def _compute_parent_requirements(targets: dict[str, int]) -> dict[str, int]:
+ """For each parent table, return the minimum row count needed so that
+ parent1 × parent2 ≥ target for every junction it participates in.
+
+ Allocates ceil(sqrt(target)) rows per parent, balanced across the two
+ parents of each junction. The actual junction seeder will then walk
+ the cross-product to produce the target number of unique pairs.
+ """
+ parent_req: dict[str, int] = {}
+ for junction, _, _, p1, p2 in JUNCTIONS:
+ target = targets.get(junction, 0)
+ if target == 0:
+ continue
+ sqrt_n = int(target**0.5) + 1
+ parent_req[p1] = max(parent_req.get(p1, 0), sqrt_n)
+ parent_req[p2] = max(parent_req.get(p2, 0), sqrt_n)
+ return parent_req
+
+
+def _seed_parents(conn: Connection, parent_req: dict[str, int], dry_run: bool) -> None:
+ """Seed parent tables in dependency order:
+ independent parents (ab_user, ab_role) first, then dashboards / slices /
+ tables (which transitively depend on dbs, seeded inside seed_tables)."""
+ if "ab_user" in parent_req:
+ seed_users(conn, parent_req["ab_user"], dry_run)
+ if "ab_role" in parent_req:
+ seed_roles(conn, parent_req["ab_role"], dry_run)
+ if "dashboards" in parent_req:
+ seed_dashboards(conn, parent_req["dashboards"], dry_run)
+ if "slices" in parent_req:
+ seed_slices(conn, parent_req["slices"], dry_run)
+ if "tables" in parent_req:
+ seed_tables(conn, parent_req["tables"], dry_run)
+
+
+def _seed_all_junctions(
+ conn: Connection, targets: dict[str, int], dry_run: bool
+) -> None:
+ for junction, fk1, fk2, p1, p2 in JUNCTIONS:
+ target = targets.get(junction, 0)
+ if target == 0:
+ continue
+ with time_phase(f"junction:{junction}"):
+ seed_junction(conn, junction, fk1, fk2, p1, p2, target, dry_run)
+
+
+def inject_duplicates(
+ conn: Connection,
+ junction: str,
+ fk1_col: str,
+ fk2_col: str,
+ pct: float,
+ dry_run: bool,
+) -> None:
+ """Insert duplicate ``(fk1, fk2)`` rows on a non-UNIQUE junction table.
+
+ Used to stress-test the migration's ``_dedupe_by_min_id`` phase, which
+ is otherwise a no-op on cleanly-seeded data. Computes ``count =
+ current_rows * pct / 100`` and inserts that many rows by re-sampling
+ existing ``(fk1, fk2)`` pairs in row-major order. The synthetic
+ duplicates land on top of distinct existing pairs (one duplicate per
+ distinct pair, then wraps), so the migration's dedupe finds and
+ deletes them.
+
+ **Pre-condition: the table must NOT have UNIQUE on (fk1, fk2)**, i.e.,
+ the schema must be the pre-migration shape (after running
+ ``superset db downgrade``). On the post-migration schema the composite
+ PK rejects duplicates and this function will error.
+ """
+ if pct == 0:
+ return
+ current = count_rows(conn, junction)
+ count = int(current * pct / 100)
+ if count == 0:
+ logger.info(
+ "%s: 0 duplicates to inject (current=%d, pct=%g)",
+ junction,
+ current,
+ pct,
+ )
+ return
+ logger.info(
+ "%s: injecting %d duplicate rows (%g%% of %d existing)",
+ junction,
+ count,
+ pct,
+ current,
+ )
+ if dry_run:
+ return
+
+ select_sql = sa.text(
+ f"SELECT {fk1_col}, {fk2_col} FROM {junction} ORDER BY id LIMIT :n" # noqa: S608
+ )
+ sample = conn.execute(select_sql, {"n": count}).fetchall()
+ if not sample:
+ logger.warning("%s: no rows to duplicate (table is empty)", junction)
+ return
+
+ insert_sql = sa.text(
+ f"INSERT INTO {junction} ({fk1_col}, {fk2_col}) " # noqa: S608
+ f"VALUES (:fk1, :fk2)"
+ )
+ inserted = 0
+ while inserted < count:
+ batch: list[dict[str, int]] = []
+ while len(batch) < BATCH and inserted < count:
+ row = sample[inserted % len(sample)]
+ batch.append({"fk1": row[0], "fk2": row[1]})
+ inserted += 1
+ conn.execute(insert_sql, batch)
+ logger.info(" %s: injected %d / %d duplicates", junction, inserted, count)
+
+
+def _inject_dirty_data(conn: Connection, dirty_pct: float, dry_run: bool) -> None:
+ """Inject duplicate rows on every non-UNIQUE seeded junction.
+
+ The two tables that originally carried ``UNIQUE(fk1, fk2)`` are
+ skipped because their composite-PK successor (and their pre-migration
+ UNIQUE constraint) both reject duplicate inserts.
+ """
+ if dirty_pct == 0:
+ return
+ for junction, fk1, fk2, _, _ in JUNCTIONS:
+ if junction in JUNCTIONS_WITH_UNIQUE:
+ logger.info(
+ "%s: skipping duplicate injection (table has UNIQUE on FK pair)",
+ junction,
+ )
+ continue
+ with time_phase(f"dirty:{junction}"):
+ inject_duplicates(conn, junction, fk1, fk2, dirty_pct, dry_run)
+
+
+def run(targets: dict[str, int], dry_run: bool, dirty_duplicates_pct: float) -> None:
+ engine = build_engine()
+ with engine.begin() as conn:
+ parent_req = _compute_parent_requirements(targets)
+ logger.info("Required parent row counts: %s", parent_req)
+
+ with time_phase("parents"):
+ _seed_parents(conn, parent_req, dry_run)
+
+ with time_phase("junctions"):
+ _seed_all_junctions(conn, targets, dry_run)
+
+ if dirty_duplicates_pct > 0:
+ with time_phase("dirty-duplicates"):
+ _inject_dirty_data(conn, dirty_duplicates_pct, dry_run)
+
+
+# ----------------------------------------------------------------------
+# CLI
+# ----------------------------------------------------------------------
+
+
+def main() -> None:
+ parser = argparse.ArgumentParser(
+ description=__doc__,
+ formatter_class=argparse.RawDescriptionHelpFormatter,
+ )
+ for table, default in DEFAULTS.items():
+ parser.add_argument(
+ f"--{table.replace('_', '-')}",
+ type=int,
+ default=default,
+ help=f"target row count for {table} (default: {default:,})",
+ )
+ parser.add_argument(
+ "--dry-run",
+ "-n",
+ action="store_true",
+ help="print planned inserts without writing to the DB",
+ )
+ parser.add_argument(
+ "--dirty-duplicates-pct",
+ type=float,
+ default=0,
+ help=(
+ "after seeding distinct pairs, inject this percentage of duplicate "
+ "rows on each non-UNIQUE junction (slice_user, dashboard_user, "
+ "dashboard_roles). Stress-tests the migration's _dedupe_by_min_id "
+ "phase. Requires 2bee73611e32 to NOT be applied: un-apply it by "
+ "downgrading to its parent (`superset db downgrade <down_revision>`, "
+ "where <down_revision> is read from the 2bee73611e32 migration "
+ "file) — the post-migration composite PK rejects duplicates and "
+ "this will error. Default: 0 (no duplicates)."
+ ),
+ )
+ parser.add_argument(
+ "--verbose",
+ "-v",
+ action="store_true",
+ help="increase log verbosity",
+ )
+ args = parser.parse_args()
+
+ logging.basicConfig(
+ level=logging.DEBUG if args.verbose else logging.INFO,
+ format="%(asctime)s [%(levelname)s] %(message)s",
+ datefmt="%H:%M:%S",
+ )
+
+ targets = {table: getattr(args, table) for table in DEFAULTS}
Review Comment:
**Suggestion:** Add an explicit type annotation for this computed mapping so the variable’s key/value types are declared. [custom_rule]
**Severity Level:** Minor ⚠️
<details>
<summary><b>Why it matters? 🤔 </b></summary>
The local variable `targets` is a derived mapping with an obvious `dict[str, int]` shape, but it is assigned without an explicit annotation, which is exactly the kind of omission the rule flags.
</details>
[](https://app.codeant.ai/fix-in-ide?tool=cursor&prompt_id=6110dee3d3fe4a2cbc061e5f6cd7fa2a&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset) [](https://app.codeant.ai/fix-in-ide?tool=vscode-claude&prompt_id=6110dee3d3fe4a2cbc061e5f6cd7fa2a&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset)
*(Use Cmd/Ctrl + Click for best experience)*
<details>
<summary><b>Prompt for AI Agent 🤖 </b></summary>
```mdx
This is a comment left during a code review.
**Path:** scripts/seed_junction_load.py
**Line:** 664:664
**Comment:**
*Custom Rule: Add an explicit type annotation for this computed mapping so the variable’s key/value types are declared.
Validate the correctness of the flagged issue. If correct, How can I resolve this? If you propose a fix, implement it and please make it concise.
Once fix is implemented, also check other comments on the same PR, and ask user if the user wants to fix the rest of the comments as well. if said yes, then fetch all the comments validate the correctness and implement a minimal fix
```
</details>
<a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F41076&comment_hash=b6c40ca1b921657754596c67862b5ceef35ad248bdbbda3e24f62e2639c8e587&reaction=like'>👍</a> | <a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F41076&comment_hash=b6c40ca1b921657754596c67862b5ceef35ad248bdbbda3e24f62e2639c8e587&reaction=dislike'>👎</a>
##########
superset/commands/dashboard/update.py:
##########
@@ -59,23 +59,31 @@ def __init__(self, model_id: int, data: dict[str, Any]):
def run(self) -> Model:
self.validate()
assert self._model is not None
- self.process_tab_diff()
- self.process_native_filter_diff()
-
- # Update tags
- if (tags := self._properties.pop("tags", None)) is not None:
- update_tags(ObjectType.dashboard, self._model.id, self._model.tags, tags)
-
- # Re-serialize position_json to escape 4-byte Unicode characters
- if position_json := self._properties.get("position_json"):
- self._properties["position_json"] = json.dumps(json.loads(position_json))
-
- dashboard = DashboardDAO.update(self._model, self._properties)
- if self._properties.get("json_metadata"):
- DashboardDAO.set_dash_metadata(
- dashboard,
- data=json.loads(self._properties.get("json_metadata", "{}")),
- )
+ # Suppress autoflush during the update body so that Continuum's
+ # before_flush baseline listener does not fire mid-operation while
+ # the session is only partially populated.
+ with db.session.no_autoflush:
+ self.process_tab_diff()
+ self.process_native_filter_diff()
+
+ # Update tags
+ if (tags := self._properties.pop("tags", None)) is not None:
+ update_tags(
+ ObjectType.dashboard, self._model.id, self._model.tags, tags
+ )
+
+ # Re-serialize position_json to escape 4-byte Unicode characters
+ if position_json := self._properties.get("position_json"):
+ self._properties["position_json"] = json.dumps(
+ json.loads(position_json)
+ )
+
+ dashboard = DashboardDAO.update(self._model, self._properties)
Review Comment:
**Suggestion:** Add an explicit type annotation for the new local variable so the modified method complies with the type-hint requirement. [custom_rule]
**Severity Level:** Minor ⚠️
<details>
<summary><b>Why it matters? 🤔 </b></summary>
The modified method introduces a new local variable without a type hint, and this variable can reasonably be annotated. That matches the stated Python type-hint rule for relevant variables in modified code.
</details>
[](https://app.codeant.ai/fix-in-ide?tool=cursor&prompt_id=1a6db024d7344836bc4bd1631c04fe2f&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset) [](https://app.codeant.ai/fix-in-ide?tool=vscode-claude&prompt_id=1a6db024d7344836bc4bd1631c04fe2f&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset)
*(Use Cmd/Ctrl + Click for best experience)*
<details>
<summary><b>Prompt for AI Agent 🤖 </b></summary>
```mdx
This is a comment left during a code review.
**Path:** superset/commands/dashboard/update.py
**Line:** 81:81
**Comment:**
*Custom Rule: Add an explicit type annotation for the new local variable so the modified method complies with the type-hint requirement.
Validate the correctness of the flagged issue. If correct, How can I resolve this? If you propose a fix, implement it and please make it concise.
Once fix is implemented, also check other comments on the same PR, and ask user if the user wants to fix the rest of the comments as well. if said yes, then fetch all the comments validate the correctness and implement a minimal fix
```
</details>
<a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F41076&comment_hash=21b38d8f89c75784df8f8a99cb9c9acb0509727c88deb8845d1c1d5092443a87&reaction=like'>👍</a> | <a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F41076&comment_hash=21b38d8f89c75784df8f8a99cb9c9acb0509727c88deb8845d1c1d5092443a87&reaction=dislike'>👎</a>
##########
superset/charts/api.py:
##########
@@ -437,9 +496,29 @@ def put(self, pk: int) -> Response:
# This validates custom Schema with custom validations
except ValidationError as error:
return self.response_400(message=error.messages)
+
+ # Live version identifiers before the update (empty + query-free when
+ # ``ENABLE_VERSIONING_CAPTURE`` is off, so this stays inert under the
+ # kill-switch).
+ old_info = current_entity_version_info(Slice, pk)
+
try:
changed_model = UpdateChartCommand(pk, item).run()
- response = self.response(200, id=changed_model.id, result=item)
+ new_info = current_entity_version_info(
+ Slice, changed_model.id, changed_model.uuid
+ )
+ response = self.response(
Review Comment:
**Suggestion:** Add an explicit type annotation to the new response local variable created in this updated block. [custom_rule]
**Severity Level:** Minor ⚠️
<details>
<summary><b>Why it matters? 🤔 </b></summary>
The updated code introduces a new local variable without a type hint. Since the variable is a concrete response object and can be annotated, this matches the type-hint requirement.
</details>
[](https://app.codeant.ai/fix-in-ide?tool=cursor&prompt_id=574d707445d740438abc4a3fb4ca6f51&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset) [](https://app.codeant.ai/fix-in-ide?tool=vscode-claude&prompt_id=574d707445d740438abc4a3fb4ca6f51&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset)
*(Use Cmd/Ctrl + Click for best experience)*
<details>
<summary><b>Prompt for AI Agent 🤖 </b></summary>
```mdx
This is a comment left during a code review.
**Path:** superset/charts/api.py
**Line:** 510:510
**Comment:**
*Custom Rule: Add an explicit type annotation to the new response local variable created in this updated block.
Validate the correctness of the flagged issue. If correct, How can I resolve this? If you propose a fix, implement it and please make it concise.
Once fix is implemented, also check other comments on the same PR, and ask user if the user wants to fix the rest of the comments as well. if said yes, then fetch all the comments validate the correctness and implement a minimal fix
```
</details>
<a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F41076&comment_hash=6d8d71598a4319f3ce9ac5d2617a19c104d453756e0040560de6757aeebcbae8&reaction=like'>👍</a> | <a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F41076&comment_hash=6d8d71598a4319f3ce9ac5d2617a19c104d453756e0040560de6757aeebcbae8&reaction=dislike'>👎</a>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(versioning): cross-entity version activity view [superset]
Posted by "bito-code-review[bot] (via GitHub)" <gi...@apache.org>.
bito-code-review[bot] commented on PR #41076:
URL: https://github.com/apache/superset/pull/41076#issuecomment-4712516079
AI Code Review is in progress (usually takes 3 to 15 minutes unless it's a very large PR).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(versioning): cross-entity version activity view [superset]
Posted by "rusackas (via GitHub)" <gi...@apache.org>.
rusackas merged PR #41076:
URL: https://github.com/apache/superset/pull/41076
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(versioning): cross-entity version activity view [superset]
Posted by "codeant-ai-for-open-source[bot] (via GitHub)" <gi...@apache.org>.
codeant-ai-for-open-source[bot] commented on code in PR #41076:
URL: https://github.com/apache/superset/pull/41076#discussion_r3560230550
##########
superset/versioning/activity/kinds.py:
##########
@@ -0,0 +1,210 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Kind translation tables, shared types, and the shadow-model loader.
+
+The activity-view module operates in two "kind" namespaces — the
+table-stored value (``"chart"`` / ``"dashboard"`` / ``"dataset"``) that
+``version_changes.entity_kind`` carries, and the Python class-name
+form (``"Slice"`` / ``"Dashboard"`` / ``"SqlaTable"``) used internally
+for dispatch and returned in the DTO's ``entity_kind`` field. The four
+mappings here translate between them. Adjacent kind-keyed dicts live
+here too: the per-kind human-readable label, the user-facing
+lowercase form, and the 404 exception class.
+
+The :func:`load_live_model` helper exists in the same module
+because each lookup is keyed on the same set of class names — keeping
+it adjacent to the mappings makes the kind-translation surface
+discoverable at a glance.
+"""
+
+from __future__ import annotations
+
+from collections.abc import Iterable, Iterator
+from dataclasses import dataclass
+
+from flask_appbuilder import Model
+
+from superset.commands.chart.exceptions import ChartNotFoundError
+from superset.commands.dashboard.exceptions import DashboardNotFoundError
+from superset.commands.dataset.exceptions import DatasetNotFoundError
+from superset.versioning.changes import ENTITY_KIND_BY_CLASS_NAME
+
+# Max entity ids per ``IN (...)`` clause across the activity-view queries.
+# Bounds the bind-parameter count under SQLite's ``SQLITE_MAX_VARIABLE_NUMBER``
+# floor (default 999 in many builds); Postgres/MySQL accept the full list but
+# the chunk is dialect-agnostic. 500 leaves headroom for the other bound
+# params on each statement.
+ENTITY_ID_CHUNK_SIZE = 500
+
+
+def chunked_ids(
+ ids: Iterable[int], size: int = ENTITY_ID_CHUNK_SIZE
+) -> Iterator[list[int]]:
+ """Yield *ids* in fixed-size lists (final chunk may be smaller). Shared by
+ the activity-view query helpers so every ``IN (...)`` stays under the
+ SQLite bind-variable floor."""
+ items = list(ids)
+ for i in range(0, len(items), size):
+ yield items[i : i + size]
+
+
+# ---- Kind translation -----------------------------------------------------
+
+# ``version_changes.entity_kind`` stores the friendly downstream-tooling
+# value (``"chart"``, ``"dashboard"``, ``"dataset"``) per the
+# ``ENTITY_KIND_BY_CLASS_NAME``. The activity-view DTO returns the
+# Python class name instead (``"Slice"``, ``"Dashboard"``,
+# ``"SqlaTable"``) so the contract aligns with ``__class__.__name__``
+# (the ``ActivityRecord`` DTO). Translate at the boundary.
+TABLE_KIND_TO_API: dict[str, str] = {
+ table_kind: class_name
+ for class_name, table_kind in ENTITY_KIND_BY_CLASS_NAME.items()
+}
+API_KIND_TO_TABLE: dict[str, str] = dict(ENTITY_KIND_BY_CLASS_NAME)
+
+# Human-readable label for summary headlines
+# ("Dataset updated: Sales Transactions"). Keyed by the internal API kind
+# (Python class name; matches ``model_cls.__name__``).
+API_KIND_LABEL: dict[str, str] = {
+ "Dashboard": "Dashboard",
+ "Slice": "Chart",
+ "SqlaTable": "Dataset",
+}
+
+# User-facing lowercase rendering of the kind. This is what appears in
+# the JSON response's ``entity_kind`` field and the
+# ``ActivityRecordSchema.entity_kind`` enum. Internal code keeps the
+# Python class-name form because it matches ``model_cls.__name__`` and is
+# convenient for dispatch — translation happens at serialization time
+# only, in :func:`render.apply_record_decoration`.
+USER_FACING_KIND: dict[str, str] = {
+ "Dashboard": "dashboard",
+ "Slice": "chart",
+ "SqlaTable": "dataset",
+}
+
+# 404 exception class per API kind. Each accepts a string positional arg
+# (the path-entity UUID) that gets formatted into the exception message.
+NOT_FOUND_EXC: dict[str, type[Exception]] = {
+ "Dashboard": DashboardNotFoundError,
+ "Slice": ChartNotFoundError,
+ "SqlaTable": DatasetNotFoundError,
+}
+
+# Per-API-kind (model class name, display column) used by
+# ``_resolve_names_for_kind`` to read the user-facing entity name from
+# the shadow table valid at a given transaction.
+NAME_COLUMN: dict[str, tuple[str, str]] = {
+ "Dashboard": ("Dashboard", "dashboard_title"),
+ "Slice": ("Slice", "slice_name"),
+ "SqlaTable": ("SqlaTable", "table_name"),
+}
+
+
+# ---- Types ----------------------------------------------------------------
+
+
+@dataclass(frozen=True)
+class Window:
+ """A validity window in Continuum transaction-id space, half-open as
+ ``[start_tx, end_tx)``.
+
+ Immutable and equal-by-attributes — two windows with the same
+ ``start_tx`` / ``end_tx`` are interchangeable. Constructor rejects
+ ``end_tx <= start_tx``. ``end_tx = None`` means "open ended
+ (current)" and acts as positive infinity throughout the helpers.
+
+ Helper methods (``contains`` / ``intersect`` / ``merges_with``)
+ live on the type so callers don't re-implement the half-open
+ predicate inline. Previously a ``tuple[int, int | None]`` alias;
+ promoted to a dataclass so a function accepting a ``Window`` can't
+ silently accept any other 2-tuple and so the constructor enforces
+ the half-open invariant.
+ """
+
+ start_tx: int
+ end_tx: int | None
+
+ def __post_init__(self) -> None:
+ if self.end_tx is not None and self.end_tx <= self.start_tx:
+ raise ValueError(
+ f"Window end_tx must be > start_tx; "
+ f"got [{self.start_tx}, {self.end_tx})"
+ )
+
+ def contains(self, tx_id: int) -> bool:
+ """``True`` iff *tx_id* falls inside this half-open interval."""
+ return self.start_tx <= tx_id and (self.end_tx is None or tx_id < self.end_tx)
+
+ def intersect(self, other: Window) -> Window | None:
+ """Return the clipped overlap of this window with *other*, or
+ ``None`` when they are disjoint. ``end_tx = None`` acts as
+ positive infinity on either side."""
+ start = max(self.start_tx, other.start_tx)
+ end: int | None
+ if self.end_tx is None:
+ end = other.end_tx
+ elif other.end_tx is None:
+ end = self.end_tx
+ else:
+ end = min(self.end_tx, other.end_tx)
+ if end is not None and end <= start:
+ return None
+ return Window(start, end)
+
+ def merges_with(self, other: Window) -> bool:
+ """``True`` iff *self* and *other* overlap or touch (so their
+ union is one contiguous window). Assumes the caller has placed
+ them in start-ascending order."""
+ if self.end_tx is None:
+ # self extends to +∞; everything past it merges in.
+ return True
+ return other.start_tx <= self.end_tx
+
+
+#: A related-entity scope row: ``(api_kind, entity_id, [windows])``.
+#: ``api_kind`` is the DTO-facing kind (``"Slice"``, etc.), not the
+#: table-stored kind. Modelled as a tuple alias — the
+#: ``(api_kind, entity_id)`` pair is logically a key with the window
+#: list as its value, so a dict shape may fit better than a flat
+#: dataclass.
+EntityWindows = tuple[str, int, list[Window]]
+
+
+def load_live_model(model_name: str) -> type[Model]:
+ """Inline-import a live model class by name. Deferred until call
+ time because the versioning package is initialised before all model
+ mappers are configured (same idiom used throughout
+ :mod:`superset.versioning.changes`).
+
+ Returns the LIVE model class (``Dashboard`` / ``Slice`` /
+ ``SqlaTable``), not a Continuum shadow class — callers derive the
+ shadow table separately via ``version_class(...)`` where needed."""
+ # pylint: disable=import-outside-toplevel
+ if model_name == "Dashboard":
+ from superset.models.dashboard import Dashboard
+
+ return Dashboard
+ if model_name == "Slice":
+ from superset.models.slice import Slice
+
+ return Slice
+ if model_name == "SqlaTable":
+ from superset.connectors.sqla.models import SqlaTable
+
+ return SqlaTable
+ raise LookupError(f"No shadow class registered for {model_name!r}")
Review Comment:
**Suggestion:** The exception message says a “shadow class” is missing, but this helper loads live model classes, not shadow versions. This creates misleading runtime diagnostics when an unsupported kind is passed. Update the message to reference the live model registry so operators can accurately troubleshoot failures. [comment mismatch]
<details>
<summary><b>Severity Level:</b> Minor 🧹</summary>
```mdx
⚠️ Misleading error when activity model lookup fails.
⚠️ Operators may debug shadow registry instead of live.
```
</details>
<details>
<summary><b>Steps of Reproduction ✅ </b></summary>
```mdx
1. Open a Python shell in the Superset environment with this PR code deployed.
2. Import the helper from `superset/versioning/activity/kinds.py` (defined at lines
188-197): `from superset.versioning.activity.kinds import load_live_model`.
3. Call `load_live_model("FooModel")`, where `"FooModel"` is not one of `"Dashboard"`,
`"Slice"`, or `"SqlaTable"` (the supported live model names wired in the `if` branches at
lines 199-208).
4. Observe that `load_live_model` raises `LookupError` at
`superset/versioning/activity/kinds.py:210` with message `No shadow class registered for
'FooModel'`, even though the function’s docstring (lines 188-197) states it returns live
model classes and callers derive shadow tables separately via `version_class(...)`; this
mismatch makes runtime diagnostics misleading about which registry is at fault.
```
</details>
[](https://app.codeant.ai/fix-in-ide?tool=cursor&prompt_id=575177774e0b48b38051220cc77bb8bc&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset) [](https://app.codeant.ai/fix-in-ide?tool=vscode-claude&prompt_id=575177774e0b48b38051220cc77bb8bc&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset)
*(Use Cmd/Ctrl + Click for best experience)*
<details>
<summary><b>Prompt for AI Agent 🤖 </b></summary>
```mdx
This is a comment left during a code review.
**Path:** superset/versioning/activity/kinds.py
**Line:** 210:210
**Comment:**
*Comment Mismatch: The exception message says a “shadow class” is missing, but this helper loads live model classes, not shadow versions. This creates misleading runtime diagnostics when an unsupported kind is passed. Update the message to reference the live model registry so operators can accurately troubleshoot failures.
Validate the correctness of the flagged issue. If correct, How can I resolve this? If you propose a fix, implement it and please make it concise.
Once fix is implemented, also check other comments on the same PR, and ask user if the user wants to fix the rest of the comments as well. if said yes, then fetch all the comments validate the correctness and implement a minimal fix
```
</details>
<a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F41076&comment_hash=bb47ad8365c36b4997a9f0d3ef3048a74f58868b136e12c108a65a89fbea5e97&reaction=like'>👍</a> | <a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F41076&comment_hash=bb47ad8365c36b4997a9f0d3ef3048a74f58868b136e12c108a65a89fbea5e97&reaction=dislike'>👎</a>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(versioning): cross-entity version activity view [superset]
Posted by "bito-code-review[bot] (via GitHub)" <gi...@apache.org>.
bito-code-review[bot] commented on PR #41076:
URL: https://github.com/apache/superset/pull/41076#issuecomment-4846986774
AI Code Review is in progress (usually takes 3 to 15 minutes unless it's a very large PR).
<details>
<summary>
Bito Usage Guide
</summary>
**Commands**
Type the following command in the pull request comment and save the comment.
- `/review` - Manually triggers a full AI review.
- `/pause` - Pauses automatic reviews on this pull request.
- `/resume` - Resumes automatic reviews.
- `/resolve` - Marks all Bito-posted review comments as resolved.
- `/abort` - Cancels all in-progress reviews.
Refer to the <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/available-commands" target="_blank" rel="noopener noreferrer">documentation</a> for additional commands.
**Configuration**
This repository uses `Superset` You can customize the agent settings <a href="https://alpha.bito.ai/home/ai-agents/code-review-agent" target="_blank" rel="noopener noreferrer">here</a> or contact your Bito workspace admin at evan@preset.io.
**Documentation & Help**
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/getting-started/install-run-using-bito-cloud/create-or-customize-an-agent-instance" target="_blank" rel="noopener noreferrer">Customize agent settings</a>
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/implementing-custom-code-review-rules" target="_blank" rel="noopener noreferrer">Review rules</a>
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent" target="_blank" rel="noopener noreferrer">General documentation</a>
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/faqs" target="_blank" rel="noopener noreferrer">FAQ</a>
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(versioning): cross-entity version activity view [superset]
Posted by "codeant-ai-for-open-source[bot] (via GitHub)" <gi...@apache.org>.
codeant-ai-for-open-source[bot] commented on code in PR #41076:
URL: https://github.com/apache/superset/pull/41076#discussion_r3500164244
##########
superset/initialization/__init__.py:
##########
@@ -619,6 +619,181 @@ def init_extensions(self) -> None:
extension.manifest.id,
)
+ @staticmethod
+ def _remove_continuum_write_listeners() -> None:
+ """Detach SQLAlchemy-Continuum's own write listeners.
+
+ ``make_versioned()`` runs unconditionally at import of
+ ``superset.extensions`` and registers Continuum's mapper, session,
+ and engine listeners — the ones that write shadow rows and
+ ``version_transaction`` rows on every flush. Skipping only the
+ custom baseline/change-record listeners would leave those running,
+ so with the kill-switch off the shadow tables would silently keep
+ accumulating, contradicting the documented contract.
+
+ This is deliberately a *targeted subset* of
+ ``sqlalchemy_continuum.remove_versioning()``: that helper also
+ calls ``manager.reset()``, which clears ``version_class_map`` —
+ and ``version_class()`` would then silently return the live model
+ class, breaking the read-only ``/versions/`` endpoints this flag
+ promises to keep working.
+
+ Idempotent: guarded on a representative listener so repeated app
+ initializations in one process (test fixtures) don't raise on
+ double-removal.
+ """
+ # pylint: disable=import-outside-toplevel
+ import sqlalchemy as sa
+ from sqlalchemy_continuum import versioning_manager
+
+ if not sa.event.contains(
+ sa.orm.Mapper, "after_insert", versioning_manager.track_inserts
+ ):
+ return # already detached by a prior init
+ versioning_manager.remove_operations_tracking(sa.orm.Mapper)
+ versioning_manager.remove_session_tracking(sa.orm.session.Session)
+ sa.event.remove(
+ sa.engine.Engine,
+ "before_execute",
+ versioning_manager.track_association_operations,
+ )
+ sa.event.remove(
+ sa.engine.Engine, "rollback", versioning_manager.clear_connection
+ )
+ sa.event.remove(
+ sa.engine.Engine,
+ "set_connection_execution_options",
+ versioning_manager.track_cloned_connections,
+ )
+
+ # Belt-and-suspenders: flip Continuum's master option off as well.
+ # Every write listener checks ``manager.options['versioning']`` before
+ # doing work (manager.py / unit_of_work.py), so if a future Continuum
+ # version registers an additional write listener this detach does not
+ # know to remove, that listener still no-ops. ``version_class()`` reads
+ # from ``version_class_map`` and ignores this option, so the read-only
+ # ``/versions/`` endpoints are unaffected.
+ versioning_manager.options["versioning"] = False
+
+ # Verify the known write listeners are actually gone. A Continuum
+ # upgrade that renamed a handler would make the removals above silently
+ # miss, leaving capture half-on while we report "disabled"; surface
+ # that rather than booting in a contradictory state.
+ if sa.event.contains(
+ sa.orm.Mapper, "after_insert", versioning_manager.track_inserts
+ ):
+ logger.warning(
+ "versioning: Continuum write listeners still attached after "
+ "detach; capture may not be fully disabled. This usually means "
+ "the pinned sqlalchemy-continuum version changed how it "
+ "registers listeners."
+ )
+
+ def init_versioning(self) -> None:
+ """Register SQLAlchemy-Continuum baseline and retention listeners.
Review Comment:
**Suggestion:** The docstring says this method registers baseline and retention listeners, but the implementation registers baseline and change-record listeners and explicitly states retention is no longer synchronous here. This mismatch is misleading for operators and maintainers; update the docstring to describe the actual listeners registered. [docstring mismatch]
<details>
<summary><b>Severity Level:</b> Minor 🧹</summary>
```mdx
- ⚠️ Docstring misleads about retention listener registration behavior.
- ⚠️ Could confuse debugging of version retention handling.
```
</details>
<details>
<summary><b>Steps of Reproduction ✅ </b></summary>
```mdx
1. Open `SupersetAppInitializer.init_versioning` in
`/superset/initialization/__init__.py:113-133`; the method docstring at line 693 states
`"""Register SQLAlchemy-Continuum baseline and retention listeners.` indicating it should
register baseline and retention listeners.
2. Scroll down within the same method: at lines 162-165 the code imports
`register_baseline_listener` and `VERSIONED_MODELS` from `superset.versioning.baseline`,
and at lines 179-181 it imports `register_change_record_listener` from
`superset.versioning.changes`, but there is no import or reference to any retention
listener.
3. At the end of `init_versioning`, the implementation at lines 210-211 calls
`register_baseline_listener()` and `register_change_record_listener()` only, and
immediately below, lines 213-216 explicitly document that "Retention pruning runs
out-of-band as a scheduled Celery beat task" and that the previous synchronous retention
listener was retired, confirming that retention listeners are no longer registered here.
4. Because the docstring still claims this method registers "baseline and retention
listeners" while the implementation only registers baseline and change-record listeners
and explicitly moves retention out-of-band, operators or maintainers reading the docstring
can be misled about where retention is handled, even though runtime behavior is correct.
```
</details>
[](https://app.codeant.ai/fix-in-ide?tool=cursor&prompt_id=9f90c827bde44cb9bffe18ba208e20d3&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset) [](https://app.codeant.ai/fix-in-ide?tool=vscode-claude&prompt_id=9f90c827bde44cb9bffe18ba208e20d3&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset)
*(Use Cmd/Ctrl + Click for best experience)*
<details>
<summary><b>Prompt for AI Agent 🤖 </b></summary>
```mdx
This is a comment left during a code review.
**Path:** superset/initialization/__init__.py
**Line:** 693:693
**Comment:**
*Docstring Mismatch: The docstring says this method registers baseline and retention listeners, but the implementation registers baseline and change-record listeners and explicitly states retention is no longer synchronous here. This mismatch is misleading for operators and maintainers; update the docstring to describe the actual listeners registered.
Validate the correctness of the flagged issue. If correct, How can I resolve this? If you propose a fix, implement it and please make it concise.
Once fix is implemented, also check other comments on the same PR, and ask user if the user wants to fix the rest of the comments as well. if said yes, then fetch all the comments validate the correctness and implement a minimal fix
```
</details>
<a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F41076&comment_hash=be7336d9d3c101687e0d5def56b7a88c0279137dc818acdb34bd7d1b0978eebc&reaction=like'>👍</a> | <a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F41076&comment_hash=be7336d9d3c101687e0d5def56b7a88c0279137dc818acdb34bd7d1b0978eebc&reaction=dislike'>👎</a>
##########
superset/initialization/__init__.py:
##########
@@ -619,6 +619,181 @@ def init_extensions(self) -> None:
extension.manifest.id,
)
+ @staticmethod
+ def _remove_continuum_write_listeners() -> None:
+ """Detach SQLAlchemy-Continuum's own write listeners.
+
+ ``make_versioned()`` runs unconditionally at import of
+ ``superset.extensions`` and registers Continuum's mapper, session,
+ and engine listeners — the ones that write shadow rows and
+ ``version_transaction`` rows on every flush. Skipping only the
+ custom baseline/change-record listeners would leave those running,
+ so with the kill-switch off the shadow tables would silently keep
+ accumulating, contradicting the documented contract.
+
+ This is deliberately a *targeted subset* of
+ ``sqlalchemy_continuum.remove_versioning()``: that helper also
+ calls ``manager.reset()``, which clears ``version_class_map`` —
+ and ``version_class()`` would then silently return the live model
+ class, breaking the read-only ``/versions/`` endpoints this flag
+ promises to keep working.
+
+ Idempotent: guarded on a representative listener so repeated app
+ initializations in one process (test fixtures) don't raise on
+ double-removal.
+ """
+ # pylint: disable=import-outside-toplevel
+ import sqlalchemy as sa
+ from sqlalchemy_continuum import versioning_manager
+
+ if not sa.event.contains(
+ sa.orm.Mapper, "after_insert", versioning_manager.track_inserts
+ ):
+ return # already detached by a prior init
+ versioning_manager.remove_operations_tracking(sa.orm.Mapper)
+ versioning_manager.remove_session_tracking(sa.orm.session.Session)
+ sa.event.remove(
+ sa.engine.Engine,
+ "before_execute",
+ versioning_manager.track_association_operations,
+ )
+ sa.event.remove(
+ sa.engine.Engine, "rollback", versioning_manager.clear_connection
+ )
+ sa.event.remove(
+ sa.engine.Engine,
+ "set_connection_execution_options",
+ versioning_manager.track_cloned_connections,
+ )
+
+ # Belt-and-suspenders: flip Continuum's master option off as well.
+ # Every write listener checks ``manager.options['versioning']`` before
+ # doing work (manager.py / unit_of_work.py), so if a future Continuum
+ # version registers an additional write listener this detach does not
+ # know to remove, that listener still no-ops. ``version_class()`` reads
+ # from ``version_class_map`` and ignores this option, so the read-only
+ # ``/versions/`` endpoints are unaffected.
+ versioning_manager.options["versioning"] = False
+
+ # Verify the known write listeners are actually gone. A Continuum
+ # upgrade that renamed a handler would make the removals above silently
+ # miss, leaving capture half-on while we report "disabled"; surface
+ # that rather than booting in a contradictory state.
+ if sa.event.contains(
+ sa.orm.Mapper, "after_insert", versioning_manager.track_inserts
+ ):
+ logger.warning(
+ "versioning: Continuum write listeners still attached after "
+ "detach; capture may not be fully disabled. This usually means "
+ "the pinned sqlalchemy-continuum version changed how it "
+ "registers listeners."
+ )
+
+ def init_versioning(self) -> None:
+ """Register SQLAlchemy-Continuum baseline and retention listeners.
+
+ Must be called after all versioned model classes have been imported so
+ that VERSIONED_MODELS can be populated and configure_mappers() has run.
+
+ ``ENABLE_VERSIONING_CAPTURE`` (ships default ``False``) gates the two
+ before-flush listener registrations. The flag is operational, not
+ feature: with it off the infrastructure is inert (no save writes
+ shadow rows); flipping it on activates capture. The switch also lets
+ an operator who observes a versioning-induced regression (e.g. a
+ save-path slowdown attributable to the change-record listener)
+ disable capture in ``superset_config.py`` and restart workers — a
+ 30-second recovery instead of revert-and-redeploy. Shadow tables
+ already created by the migration stay; they just stop accumulating
+ new rows.
+
+ The fallback here is ``False`` so that any app-factory path that
+ does not load ``superset.config`` (some test factories, embedded
+ use) stays inert by default rather than silently enabling capture.
+ """
+ if not self.config.get("ENABLE_VERSIONING_CAPTURE", False):
+ logger.warning(
+ "versioning: ENABLE_VERSIONING_CAPTURE is False; "
+ "skipping baseline + change-record listener registration "
+ "and detaching Continuum's write listeners. Save-path "
+ "capture is disabled; existing shadow tables and "
+ "/versions/ endpoints continue to work read-only."
+ )
+ self._remove_continuum_write_listeners()
+ return
+
+ # Symmetric with the OFF branch's ``options['versioning'] = False``:
+ # re-assert it on here so capture is restored even if a prior app
+ # init in the same process (multi-app / test reentrancy) flipped the
+ # process-global Continuum option off. Without this, an OFF app
+ # initialized before an ON app would leave the option False and the
+ # baseline listener — which gates on it — would silently write no
+ # baselines despite capture being "enabled".
+ from sqlalchemy_continuum import versioning_manager
+
+ versioning_manager.options["versioning"] = True
Review Comment:
**Suggestion:** The OFF path detaches Continuum's mapper/session/engine write listeners, but the ON path only flips `versioning_manager.options["versioning"]` back to `True` and never re-attaches those listeners. In a process that initializes with capture OFF and later ON (multi-app/test reentrancy), version rows will silently stop being written even though capture is reported as enabled. Re-register the Continuum listeners when enabling capture (or avoid physical listener removal and rely on the option gate only). [incomplete implementation]
<details>
<summary><b>Severity Level:</b> Major ⚠️</summary>
```mdx
- ❌ Version history not recorded after OFF-then-ON initialization.
- ⚠️ Activity view endpoints miss changes after config toggle.
- ⚠️ Multi-app tests see inconsistent version capture behavior.
```
</details>
<details>
<summary><b>Steps of Reproduction ✅ </b></summary>
```mdx
1. In a fresh Python process, import `superset.extensions`, which constructs `db` and
calls `make_versioned()` in `/superset/extensions/__init__.py:29-53`, attaching
SQLAlchemy-Continuum's mapper/session/engine write listeners for versioning to the global
`versioning_manager`.
2. Instantiate a `SupersetAppInitializer` whose `config["ENABLE_VERSIONING_CAPTURE"]` is
`False` and call `init_app_in_ctx()` in `/superset/initialization/__init__.py:218-246`,
which calls `self.init_versioning()`; the False branch at lines 713-143 logs a warning,
then calls `_remove_continuum_write_listeners()` at lines 43-88, which invokes
`versioning_manager.remove_operations_tracking`, `remove_session_tracking`, and several
`sa.event.remove(...)` calls to detach Continuum's write listeners and sets
`versioning_manager.options["versioning"] = False`.
3. In the same Python process (multi-app or test reentrancy), instantiate a second
`SupersetAppInitializer` with `config["ENABLE_VERSIONING_CAPTURE"]` set to `True` and call
`init_app_in_ctx()` again; this time `init_versioning()` at lines 145-155 only imports
`versioning_manager` and sets `versioning_manager.options["versioning"] = True`, then
registers the baseline and change-record listeners via `register_baseline_listener()` and
`register_change_record_listener()` at lines 210-211, but nowhere in the codebase
(verified via Grep for `make_versioned` and `remove_operations_tracking`) are Continuum's
mapper/session/engine listeners re-attached after `_remove_continuum_write_listeners()`
has removed them.
4. Perform any write to a versioned model, e.g. update a dataset through the dataset
update flow in `/superset/datasets/api.py:509-513` (which depends on versioning via
`current_entity_version_info`) or similar dashboard/chart update flows in
`/superset/dashboards/api.py:892-894` and `/superset/charts/api.py:501-503`; because
Continuum's write listeners are still detached despite `ENABLE_VERSIONING_CAPTURE` being
True, no new shadow rows or `version_transaction` rows are written, so change-record
capture in `/superset/versioning/changes/listener.py:29-37` and activity queries in
`/superset/versioning/activity/queries.py:27-40` silently see missing version history even
though capture is reported as enabled.
```
</details>
[](https://app.codeant.ai/fix-in-ide?tool=cursor&prompt_id=6e4044d2cb4245d9a2a5a47052a725cf&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset) [](https://app.codeant.ai/fix-in-ide?tool=vscode-claude&prompt_id=6e4044d2cb4245d9a2a5a47052a725cf&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset)
*(Use Cmd/Ctrl + Click for best experience)*
<details>
<summary><b>Prompt for AI Agent 🤖 </b></summary>
```mdx
This is a comment left during a code review.
**Path:** superset/initialization/__init__.py
**Line:** 713:733
**Comment:**
*Incomplete Implementation: The OFF path detaches Continuum's mapper/session/engine write listeners, but the ON path only flips `versioning_manager.options["versioning"]` back to `True` and never re-attaches those listeners. In a process that initializes with capture OFF and later ON (multi-app/test reentrancy), version rows will silently stop being written even though capture is reported as enabled. Re-register the Continuum listeners when enabling capture (or avoid physical listener removal and rely on the option gate only).
Validate the correctness of the flagged issue. If correct, How can I resolve this? If you propose a fix, implement it and please make it concise.
Once fix is implemented, also check other comments on the same PR, and ask user if the user wants to fix the rest of the comments as well. if said yes, then fetch all the comments validate the correctness and implement a minimal fix
```
</details>
<a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F41076&comment_hash=3855e41e4d7341b6326cdb5f471d7bf1d13c7583f477e136d51ca25598192f43&reaction=like'>👍</a> | <a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F41076&comment_hash=3855e41e4d7341b6326cdb5f471d7bf1d13c7583f477e136d51ca25598192f43&reaction=dislike'>👎</a>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(versioning): cross-entity version activity view [superset]
Posted by "bito-code-review[bot] (via GitHub)" <gi...@apache.org>.
bito-code-review[bot] commented on PR #41076:
URL: https://github.com/apache/superset/pull/41076#issuecomment-4938473833
AI Code Review is in progress (usually takes 3 to 15 minutes unless it's a very large PR).
<details>
<summary>
Bito Usage Guide
</summary>
**Commands**
Type the following command in the pull request comment and save the comment.
- `/review` - Manually triggers a full AI review.
- `/pause` - Pauses automatic reviews on this pull request.
- `/resume` - Resumes automatic reviews.
- `/resolve` - Marks all Bito-posted review comments as resolved.
- `/abort` - Cancels all in-progress reviews.
Refer to the <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/available-commands" target="_blank" rel="noopener noreferrer">documentation</a> for additional commands.
**Configuration**
This repository uses `Superset` You can customize the agent settings <a href="https://alpha.bito.ai/home/ai-agents/code-review-agent" target="_blank" rel="noopener noreferrer">here</a> or contact your Bito workspace admin at evan@preset.io.
**Documentation & Help**
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/getting-started/install-run-using-bito-cloud/create-or-customize-an-agent-instance" target="_blank" rel="noopener noreferrer">Customize agent settings</a>
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/implementing-custom-code-review-rules" target="_blank" rel="noopener noreferrer">Review rules</a>
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent" target="_blank" rel="noopener noreferrer">General documentation</a>
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/faqs" target="_blank" rel="noopener noreferrer">FAQ</a>
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(versioning): cross-entity version activity view [superset]
Posted by "bito-code-review[bot] (via GitHub)" <gi...@apache.org>.
bito-code-review[bot] commented on PR #41076:
URL: https://github.com/apache/superset/pull/41076#issuecomment-4964845335
AI Code Review is in progress (usually takes 3 to 15 minutes unless it's a very large PR).
<details>
<summary>
Bito Usage Guide
</summary>
**Commands**
Type the following command in the pull request comment and save the comment.
- `/review` - Manually triggers a full AI review.
- `/pause` - Pauses automatic reviews on this pull request.
- `/resume` - Resumes automatic reviews.
- `/resolve` - Marks all Bito-posted review comments as resolved.
- `/abort` - Cancels all in-progress reviews.
Refer to the <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/available-commands" target="_blank" rel="noopener noreferrer">documentation</a> for additional commands.
**Configuration**
This repository uses `Superset` You can customize the agent settings <a href="https://alpha.bito.ai/home/ai-agents/code-review-agent" target="_blank" rel="noopener noreferrer">here</a> or contact your Bito workspace admin at evan@preset.io.
**Documentation & Help**
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/getting-started/install-run-using-bito-cloud/create-or-customize-an-agent-instance" target="_blank" rel="noopener noreferrer">Customize agent settings</a>
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/implementing-custom-code-review-rules" target="_blank" rel="noopener noreferrer">Review rules</a>
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent" target="_blank" rel="noopener noreferrer">General documentation</a>
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/faqs" target="_blank" rel="noopener noreferrer">FAQ</a>
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(versioning): cross-entity version activity view [superset]
Posted by "mikebridge (via GitHub)" <gi...@apache.org>.
mikebridge commented on code in PR #41076:
URL: https://github.com/apache/superset/pull/41076#discussion_r3575000318
##########
superset/versioning/activity/kinds.py:
##########
@@ -0,0 +1,210 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Kind translation tables, shared types, and the shadow-model loader.
+
+The activity-view module operates in two "kind" namespaces — the
+table-stored value (``"chart"`` / ``"dashboard"`` / ``"dataset"``) that
+``version_changes.entity_kind`` carries, and the Python class-name
+form (``"Slice"`` / ``"Dashboard"`` / ``"SqlaTable"``) used internally
+for dispatch and returned in the DTO's ``entity_kind`` field. The four
+mappings here translate between them. Adjacent kind-keyed dicts live
+here too: the per-kind human-readable label, the user-facing
+lowercase form, and the 404 exception class.
+
+The :func:`load_live_model` helper exists in the same module
+because each lookup is keyed on the same set of class names — keeping
+it adjacent to the mappings makes the kind-translation surface
+discoverable at a glance.
+"""
+
+from __future__ import annotations
+
+from collections.abc import Iterable, Iterator
+from dataclasses import dataclass
+
+from flask_appbuilder import Model
+
+from superset.commands.chart.exceptions import ChartNotFoundError
+from superset.commands.dashboard.exceptions import DashboardNotFoundError
+from superset.commands.dataset.exceptions import DatasetNotFoundError
+from superset.versioning.changes import ENTITY_KIND_BY_CLASS_NAME
+
+# Max entity ids per ``IN (...)`` clause across the activity-view queries.
+# Bounds the bind-parameter count under SQLite's ``SQLITE_MAX_VARIABLE_NUMBER``
+# floor (default 999 in many builds); Postgres/MySQL accept the full list but
+# the chunk is dialect-agnostic. 500 leaves headroom for the other bound
+# params on each statement.
+ENTITY_ID_CHUNK_SIZE = 500
+
+
+def chunked_ids(
+ ids: Iterable[int], size: int = ENTITY_ID_CHUNK_SIZE
+) -> Iterator[list[int]]:
+ """Yield *ids* in fixed-size lists (final chunk may be smaller). Shared by
+ the activity-view query helpers so every ``IN (...)`` stays under the
+ SQLite bind-variable floor."""
+ items = list(ids)
+ for i in range(0, len(items), size):
+ yield items[i : i + size]
+
+
+# ---- Kind translation -----------------------------------------------------
+
+# ``version_changes.entity_kind`` stores the friendly downstream-tooling
+# value (``"chart"``, ``"dashboard"``, ``"dataset"``) per the
+# ``ENTITY_KIND_BY_CLASS_NAME``. The activity-view DTO returns the
+# Python class name instead (``"Slice"``, ``"Dashboard"``,
+# ``"SqlaTable"``) so the contract aligns with ``__class__.__name__``
+# (the ``ActivityRecord`` DTO). Translate at the boundary.
+TABLE_KIND_TO_API: dict[str, str] = {
+ table_kind: class_name
+ for class_name, table_kind in ENTITY_KIND_BY_CLASS_NAME.items()
+}
+API_KIND_TO_TABLE: dict[str, str] = dict(ENTITY_KIND_BY_CLASS_NAME)
+
+# Human-readable label for summary headlines
+# ("Dataset updated: Sales Transactions"). Keyed by the internal API kind
+# (Python class name; matches ``model_cls.__name__``).
+API_KIND_LABEL: dict[str, str] = {
+ "Dashboard": "Dashboard",
+ "Slice": "Chart",
+ "SqlaTable": "Dataset",
+}
+
+# User-facing lowercase rendering of the kind. This is what appears in
+# the JSON response's ``entity_kind`` field and the
+# ``ActivityRecordSchema.entity_kind`` enum. Internal code keeps the
+# Python class-name form because it matches ``model_cls.__name__`` and is
+# convenient for dispatch — translation happens at serialization time
+# only, in :func:`render.apply_record_decoration`.
+USER_FACING_KIND: dict[str, str] = {
+ "Dashboard": "dashboard",
+ "Slice": "chart",
+ "SqlaTable": "dataset",
+}
+
+# 404 exception class per API kind. Each accepts a string positional arg
+# (the path-entity UUID) that gets formatted into the exception message.
+NOT_FOUND_EXC: dict[str, type[Exception]] = {
+ "Dashboard": DashboardNotFoundError,
+ "Slice": ChartNotFoundError,
+ "SqlaTable": DatasetNotFoundError,
+}
+
+# Per-API-kind (model class name, display column) used by
+# ``_resolve_names_for_kind`` to read the user-facing entity name from
+# the shadow table valid at a given transaction.
+NAME_COLUMN: dict[str, tuple[str, str]] = {
+ "Dashboard": ("Dashboard", "dashboard_title"),
+ "Slice": ("Slice", "slice_name"),
+ "SqlaTable": ("SqlaTable", "table_name"),
+}
+
+
+# ---- Types ----------------------------------------------------------------
+
+
+@dataclass(frozen=True)
+class Window:
+ """A validity window in Continuum transaction-id space, half-open as
+ ``[start_tx, end_tx)``.
+
+ Immutable and equal-by-attributes — two windows with the same
+ ``start_tx`` / ``end_tx`` are interchangeable. Constructor rejects
+ ``end_tx <= start_tx``. ``end_tx = None`` means "open ended
+ (current)" and acts as positive infinity throughout the helpers.
+
+ Helper methods (``contains`` / ``intersect`` / ``merges_with``)
+ live on the type so callers don't re-implement the half-open
+ predicate inline. Previously a ``tuple[int, int | None]`` alias;
+ promoted to a dataclass so a function accepting a ``Window`` can't
+ silently accept any other 2-tuple and so the constructor enforces
+ the half-open invariant.
+ """
+
+ start_tx: int
+ end_tx: int | None
+
+ def __post_init__(self) -> None:
+ if self.end_tx is not None and self.end_tx <= self.start_tx:
+ raise ValueError(
+ f"Window end_tx must be > start_tx; "
+ f"got [{self.start_tx}, {self.end_tx})"
+ )
+
+ def contains(self, tx_id: int) -> bool:
+ """``True`` iff *tx_id* falls inside this half-open interval."""
+ return self.start_tx <= tx_id and (self.end_tx is None or tx_id < self.end_tx)
+
+ def intersect(self, other: Window) -> Window | None:
+ """Return the clipped overlap of this window with *other*, or
+ ``None`` when they are disjoint. ``end_tx = None`` acts as
+ positive infinity on either side."""
+ start = max(self.start_tx, other.start_tx)
+ end: int | None
+ if self.end_tx is None:
+ end = other.end_tx
+ elif other.end_tx is None:
+ end = self.end_tx
+ else:
+ end = min(self.end_tx, other.end_tx)
+ if end is not None and end <= start:
+ return None
+ return Window(start, end)
+
+ def merges_with(self, other: Window) -> bool:
+ """``True`` iff *self* and *other* overlap or touch (so their
+ union is one contiguous window). Assumes the caller has placed
+ them in start-ascending order."""
+ if self.end_tx is None:
+ # self extends to +∞; everything past it merges in.
+ return True
+ return other.start_tx <= self.end_tx
+
+
+#: A related-entity scope row: ``(api_kind, entity_id, [windows])``.
+#: ``api_kind`` is the DTO-facing kind (``"Slice"``, etc.), not the
+#: table-stored kind. Modelled as a tuple alias — the
+#: ``(api_kind, entity_id)`` pair is logically a key with the window
+#: list as its value, so a dict shape may fit better than a flat
+#: dataclass.
+EntityWindows = tuple[str, int, list[Window]]
+
+
+def load_live_model(model_name: str) -> type[Model]:
+ """Inline-import a live model class by name. Deferred until call
+ time because the versioning package is initialised before all model
+ mappers are configured (same idiom used throughout
+ :mod:`superset.versioning.changes`).
+
+ Returns the LIVE model class (``Dashboard`` / ``Slice`` /
+ ``SqlaTable``), not a Continuum shadow class — callers derive the
+ shadow table separately via ``version_class(...)`` where needed."""
+ # pylint: disable=import-outside-toplevel
+ if model_name == "Dashboard":
+ from superset.models.dashboard import Dashboard
+
+ return Dashboard
+ if model_name == "Slice":
+ from superset.models.slice import Slice
+
+ return Slice
+ if model_name == "SqlaTable":
+ from superset.connectors.sqla.models import SqlaTable
+
+ return SqlaTable
+ raise LookupError(f"No shadow class registered for {model_name!r}")
Review Comment:
Addressed by subsequent commits; the flagged code is no longer current after the rebase onto master. Resolving as outdated.
##########
tests/integration_tests/versioning/activity_view_tests.py:
##########
@@ -0,0 +1,1250 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Integration tests for the cross-entity activity-view API (sc-107283).
+
+US1 — dashboard activity stream: ``GET /api/v1/dashboard/<uuid>/activity/``.
+Tests for US2 (chart activity) and US3 (dataset activity) come in later
+phases.
+
+Per spec T053 / sc-103156 T062, every test that mutates a fixture entity
+wraps the test body in ``try``/``finally`` with
+``metadata_db.session.rollback()`` in the ``finally``. The rationale is
+documented in the spec — Continuum captures dirty mappers during
+autoflush, so leaving an instrumented attribute dirty pollutes
+downstream tests via the shadow tables.
+"""
+
+from __future__ import annotations
+
+from typing import Any
+from uuid import uuid4
+
+import pytest
+
+from superset.connectors.sqla.models import SqlaTable
+from superset.extensions import db
+from superset.models.dashboard import Dashboard
+from superset.models.slice import Slice
+from superset.utils import json as _json
+from tests.integration_tests.base_tests import SupersetTestCase
+from tests.integration_tests.constants import (
+ ADMIN_USERNAME,
+ ALPHA_USERNAME,
+ GAMMA_USERNAME,
+)
+from tests.integration_tests.fixtures.birth_names_dashboard import ( # noqa: F401
+ load_birth_names_dashboard_with_slices,
+ load_birth_names_data,
+)
+
+
+def _get_birth_names_dataset() -> SqlaTable:
+ return (
+ db.session.query(SqlaTable)
+ .filter(SqlaTable.table_name == "birth_names")
+ .first()
+ )
+
+
+def _persist_fixture_state() -> None:
+ """Force the fixture's pending INSERTs to commit so subsequent edits
+ produce *new* version rows instead of being batched into the
+ creation transaction. Mirrors the same helper in
+ ``tests/integration_tests/dashboards/version_history_tests.py``.
+ """
+ db.session.commit()
+
+
+def _get_birth_names_dashboard() -> Dashboard:
+ return (
+ db.session.query(Dashboard)
+ .filter(Dashboard.dashboard_title == "USA Births Names")
+ .first()
+ )
+
+
+class TestDashboardActivityView(SupersetTestCase):
+ """T017–T026 — ``GET /api/v1/dashboard/<uuid>/activity/`` (US1)."""
+
+ @pytest.fixture(autouse=True)
+ def _load_data(self, load_birth_names_dashboard_with_slices): # noqa: PT004, F811
+ pass
+
+ def _activity(self, dashboard_uuid: str, **query: Any) -> Any:
+ return self.client.get(
+ f"/api/v1/dashboard/{dashboard_uuid}/activity/",
+ query_string=query,
+ )
+
+ # ---- 4xx boundary cases ----
+
+ def test_activity_returns_404_for_unknown_uuid(self) -> None:
+ """AV-009: unknown path entity → 404."""
+ self.login(ADMIN_USERNAME)
+ rv = self._activity("00000000-0000-0000-0000-000000000000")
+ assert rv.status_code == 404
+
+ def test_activity_returns_400_for_invalid_uuid(self) -> None:
+ """A malformed UUID is rejected by the endpoint, not by Werkzeug."""
+ self.login(ADMIN_USERNAME)
+ rv = self._activity("not-a-uuid")
+ assert rv.status_code == 400
+
+ def test_activity_returns_400_for_invalid_include(self) -> None:
+ _persist_fixture_state()
+ dashboard = _get_birth_names_dashboard()
+ assert dashboard is not None
+ self.login(ADMIN_USERNAME)
+ rv = self._activity(str(dashboard.uuid), include="sibling")
+ assert rv.status_code == 400
+
+ def test_activity_returns_400_for_invalid_since(self) -> None:
+ _persist_fixture_state()
+ dashboard = _get_birth_names_dashboard()
+ assert dashboard is not None
+ self.login(ADMIN_USERNAME)
+ rv = self._activity(str(dashboard.uuid), since="yesterday")
+ assert rv.status_code == 400
+
+ def test_activity_allows_read_non_owner(self) -> None:
+ """Activity is a read endpoint: a non-owner with read access (Alpha,
+ which carries broad read + datasource access) can read a dashboard's
+ activity stream — ``raise_for_access(dashboard=)`` does not reject —
+ so the endpoint returns 200. Visibility of *related* rows is filtered
+ separately, inside the activity layer."""
+ _persist_fixture_state()
+ dashboard = _get_birth_names_dashboard()
+ assert dashboard is not None
+ dashboard_uuid = str(dashboard.uuid)
+
+ self.login(ALPHA_USERNAME)
+ rv = self._activity(dashboard_uuid)
+ assert rv.status_code == 200
+
+ def test_visibility_filter_silently_drops_inaccessible_related(self) -> None:
+ """AV-008 security control: a related record whose entity the caller
+ cannot read is *silently* dropped — absent from the result, no
+ placeholder, no contribution to count. Exercises the real
+ enforcement point (``filter_records_by_visibility`` →
+ ``DashboardAccessFilter``) with a restricted Gamma principal.
+
+ Setup uses two dashboards (no datasource needed): one owned by
+ Gamma (readable), and an admin-owned one Gamma may not read.
+ """
+ from superset import security_manager
+ from superset.subjects.utils import get_user_subject
+ from superset.versioning.activity.visibility import (
+ filter_records_by_visibility,
+ )
+
+ admin = security_manager.find_user(ADMIN_USERNAME)
+ gamma = security_manager.find_user(GAMMA_USERNAME)
+ # Access control is subject-based (``editors``) since the Subject
+ # work replaced ``owners``; grant read via each user's Subject.
+ visible = Dashboard(
+ dashboard_title=f"vis-probe-owned {uuid4().hex[:8]}",
+ slug=f"vis-owned-{uuid4().hex[:8]}",
+ published=False,
+ editors=[get_user_subject(gamma.id)],
+ )
+ hidden = Dashboard(
+ dashboard_title=f"vis-probe-hidden {uuid4().hex[:8]}",
+ slug=f"vis-hidden-{uuid4().hex[:8]}",
+ published=False,
+ editors=[get_user_subject(admin.id)],
+ )
+ db.session.add_all([visible, hidden])
+ db.session.commit()
+ visible_id, hidden_id = visible.id, hidden.id
+ try:
+ records = [
+ {"entity_kind": "dashboard", "entity_id": visible_id},
+ {"entity_kind": "dashboard", "entity_id": hidden_id},
+ ]
+ self.login(GAMMA_USERNAME)
+ filtered = filter_records_by_visibility(records)
+ kept_ids = {r["entity_id"] for r in filtered}
+
+ assert visible_id in kept_ids, (
+ "gamma-owned dashboard must stay visible to Gamma"
+ )
+ assert hidden_id not in kept_ids, (
+ "unpublished admin-owned dashboard must be dropped for Gamma"
+ )
+ # Silent: the dropped record leaves nothing behind (no placeholder).
+ assert len(filtered) == 1
+ finally:
+ db.session.rollback()
+ for did in (visible_id, hidden_id):
+ obj = db.session.query(Dashboard).filter(Dashboard.id == did).first()
+ if obj is not None:
+ db.session.delete(obj)
+ db.session.commit()
+
+ # ---- 200 happy paths ----
+
+ def test_activity_returns_200_with_envelope_shape(self) -> None:
+ """Smoke test: the endpoint returns the documented envelope shape
+ (``result`` list + ``count`` integer) even when the dashboard has
+ no activity yet."""
+ _persist_fixture_state()
+ dashboard = _get_birth_names_dashboard()
+ assert dashboard is not None
+ dashboard_uuid = str(dashboard.uuid)
+
+ self.login(ADMIN_USERNAME)
+ rv = self._activity(dashboard_uuid)
+ assert rv.status_code == 200
+ body = _json.loads(rv.data.decode("utf-8"))
+ assert "result" in body
+ assert "count" in body
+ assert isinstance(body["result"], list)
+ assert isinstance(body["count"], int)
+
+ def test_activity_includes_chart_edit_as_related(self) -> None:
+ """T018 / AS-1 of US1: editing a chart on the dashboard surfaces
+ the chart-edit record with ``entity_kind=Slice`` and
+ ``source=related``."""
+ _persist_fixture_state()
+ dashboard = _get_birth_names_dashboard()
+ assert dashboard is not None
+ dashboard_uuid = str(dashboard.uuid)
+ chart_on_dashboard = next(iter(dashboard.slices), None)
+ assert chart_on_dashboard is not None
+ chart_id = chart_on_dashboard.id
+ original_name = chart_on_dashboard.slice_name
+
+ try:
+ chart_on_dashboard.slice_name = f"{original_name} (edited)"
+ db.session.commit()
+
+ self.login(ADMIN_USERNAME)
+ rv = self._activity(dashboard_uuid)
+ assert rv.status_code == 200
+ body = _json.loads(rv.data.decode("utf-8"))
+ related = [
+ r
+ for r in body["result"]
+ if r["entity_kind"] == "chart" and r["source"] == "related"
+ ]
+ assert related, (
+ "Expected at least one Slice/related record from the chart "
+ "edit; got: "
+ f"{[(r['entity_kind'], r['source']) for r in body['result']]}"
+ )
+ # Spot-check the carry-through of denormalized fields
+ sample = related[0]
+ assert sample["entity_uuid"] is not None
+ assert "transaction_id" in sample
+ assert "issued_at" in sample
+ finally:
+ db.session.rollback()
+ chart = db.session.query(Slice).filter(Slice.id == chart_id).one()
+ chart.slice_name = original_name
+ db.session.commit()
+
+ def test_activity_include_self_excludes_related(self) -> None:
+ """T023 / AV-016: ``?include=self`` filters out related records."""
+ _persist_fixture_state()
+ dashboard = _get_birth_names_dashboard()
+ assert dashboard is not None
+ dashboard_uuid = str(dashboard.uuid)
+ chart_on_dashboard = next(iter(dashboard.slices), None)
+ assert chart_on_dashboard is not None
+ chart_id = chart_on_dashboard.id
+ original_name = chart_on_dashboard.slice_name
+
+ try:
+ chart_on_dashboard.slice_name = f"{original_name} (edited self)"
+ db.session.commit()
+
+ self.login(ADMIN_USERNAME)
+ rv = self._activity(dashboard_uuid, include="self")
+ assert rv.status_code == 200
+ body = _json.loads(rv.data.decode("utf-8"))
+ for record in body["result"]:
+ assert record["source"] == "self", (
+ f"include=self leaked a non-self record: {record}"
+ )
+ assert record["entity_kind"] == "dashboard"
+ finally:
+ db.session.rollback()
+ chart = db.session.query(Slice).filter(Slice.id == chart_id).one()
+ chart.slice_name = original_name
+ db.session.commit()
+
+ def test_activity_include_related_excludes_self(self) -> None:
+ """T024 / AV-016: ``?include=related`` returns only related records."""
+ _persist_fixture_state()
+ dashboard = _get_birth_names_dashboard()
+ assert dashboard is not None
+ dashboard_uuid = str(dashboard.uuid)
+ original_title = dashboard.dashboard_title
+ dashboard_id = dashboard.id
+
+ try:
+ # Edit the dashboard's own field so we have a self record to
+ # filter out, and edit a chart on it so we have a related
+ # record to keep.
+ dashboard.dashboard_title = f"{original_title} (edited dash)"
+ db.session.commit()
+ chart_on_dashboard = next(iter(dashboard.slices), None)
+ assert chart_on_dashboard is not None
+ chart_id = chart_on_dashboard.id
+ chart_original_name = chart_on_dashboard.slice_name
+ chart_on_dashboard.slice_name = f"{chart_original_name} (edited chart)"
+ db.session.commit()
+
+ self.login(ADMIN_USERNAME)
+ rv = self._activity(dashboard_uuid, include="related")
+ assert rv.status_code == 200
+ body = _json.loads(rv.data.decode("utf-8"))
+ for record in body["result"]:
+ assert record["source"] == "related", (
+ f"include=related leaked a self record: {record}"
+ )
+ assert record["entity_kind"] != "dashboard"
+ finally:
+ db.session.rollback()
+ dashboard = (
+ db.session.query(Dashboard).filter(Dashboard.id == dashboard_id).one()
+ )
+ dashboard.dashboard_title = original_title
+ chart = db.session.query(Slice).filter(Slice.id == chart_id).one()
+ chart.slice_name = chart_original_name
+ db.session.commit()
Review Comment:
Addressed in the preceding review-fix commit: cleanup variables are initialized before the `try` and guarded before restoration.
##########
tests/integration_tests/versioning/perf_validation_tests.py:
##########
@@ -0,0 +1,416 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""T044 — Performance validation for entity version history.
+
+Skipped by default. Run on demand:
+
+ SUPERSET_PERF_VALIDATION=1 pytest \
+ tests/integration_tests/versioning/perf_validation_tests.py -v -s
+
+Measures the three success criteria defined in the spec:
+
+ * SC-002: version list endpoint responds in under 1 second
+ * SC-004: save path p95 overhead under 50 ms with Continuum tracking
+ on vs. off (FR-014)
+
+(SC-003, the restore-endpoint benchmark, lands with the restore endpoint
+in a later PR.)
+
+The test prints a summary table suitable for pasting into the PR
+description. It also asserts each target so regressions fail loudly
+when the harness is re-run.
+"""
+
+from __future__ import annotations
+
+import os
+import statistics
+import time
+from typing import Any
+
+import pytest
+import sqlalchemy as sa
+from sqlalchemy_continuum import version_class, versioning_manager
+
+from superset.extensions import db
+from superset.models.slice import Slice
+from tests.integration_tests.base_tests import SupersetTestCase
+from tests.integration_tests.constants import ADMIN_USERNAME
+from tests.integration_tests.fixtures.birth_names_dashboard import ( # noqa: F401
+ load_birth_names_dashboard_with_slices,
+ load_birth_names_data,
+)
+
+SKIP_REASON = "Performance validation is manual. Set SUPERSET_PERF_VALIDATION=1 to run."
+
+# Thresholds from spec.md §Success Criteria.
+LIST_ENDPOINT_MAX_MS = 1000 # SC-002
+SAVE_OVERHEAD_P95_MAX_MS = 50 # SC-004
+
+# Activity-view thresholds (sc-107283 §Success Criteria).
+ACTIVITY_ENDPOINT_P95_MAX_MS = 1500 # SC-AV-001
+
+
+def _save_chart_once(chart: Slice, suffix: str) -> None:
+ """One ORM-level save path, mimicking what ChartDAO.update does."""
+ chart.slice_name = f"{chart.slice_name[:64]}_{suffix}"
+ db.session.commit()
+
+
+def _timings_ms(seconds: list[float]) -> dict[str, float]:
+ ms = sorted(s * 1000.0 for s in seconds)
+ return {
+ "p50": statistics.median(ms),
+ "p95": ms[int(len(ms) * 0.95) - 1] if len(ms) >= 20 else max(ms),
+ "max": max(ms),
+ "n": len(ms),
+ }
+
+
+@pytest.mark.skipif(
+ not os.environ.get("SUPERSET_PERF_VALIDATION"),
+ reason=SKIP_REASON,
+)
+class PerfValidationTests(SupersetTestCase):
+ """Runs only when SUPERSET_PERF_VALIDATION=1 is set."""
+
+ @pytest.fixture(autouse=True)
+ def _load_data(self, load_birth_names_dashboard_with_slices: Any) -> None: # noqa: F811, PT004
+ pass
+
+ def _seed_chart_with_n_versions(self, n: int) -> Slice:
+ """Save a chart N times to produce N version rows."""
+ chart = db.session.query(Slice).first()
+ assert chart is not None, "birth_names fixture should provide charts"
+
+ for i in range(n):
+ _save_chart_once(chart, f"v{i}")
+ db.session.commit()
+ return chart
+
+ def test_sc002_list_endpoint_under_1s(self) -> None:
+ """SC-002: list endpoint responds in under 1 second."""
+ self.login(ADMIN_USERNAME)
+
+ # Generate enough versions to exercise the retention-capped state.
+ chart = self._seed_chart_with_n_versions(24)
+ chart_uuid = str(chart.uuid)
+ url = f"/api/v1/chart/{chart_uuid}/versions/"
+
+ # Warm up the endpoint once (JIT caching, mapper configuration, etc.)
+ self.client.get(url)
+
+ timings: list[float] = []
+ for _ in range(10):
+ t0 = time.perf_counter()
+ response = self.client.get(url)
+ timings.append(time.perf_counter() - t0)
+ assert response.status_code == 200
+
+ stats = _timings_ms(timings)
+ print(
+ f"\n[SC-002] GET /versions/ (24 versions) "
+ f"p50={stats['p50']:.1f}ms p95={stats['p95']:.1f}ms "
+ f"max={stats['max']:.1f}ms n={stats['n']}"
+ )
+ assert stats["p95"] < LIST_ENDPOINT_MAX_MS, (
+ f"SC-002 failed: list endpoint p95 {stats['p95']:.1f}ms "
+ f">= {LIST_ENDPOINT_MAX_MS}ms"
+ )
+
+ # SC-003 (restore endpoint under 3s) intentionally omitted here: the
+ # version-restore route (POST /<uuid>/versions/<version_uuid>/restore)
+ # ships in a later PR, so there is nothing to time at this point in the
+ # stack. Re-add this benchmark alongside the restore endpoint.
+
+ def test_sc004_save_overhead_under_50ms(self) -> None:
+ """SC-004: save path p95 overhead under 50ms (FR-014).
+
+ Toggling Continuum on and off mid-process corrupts its internal
+ ``units_of_work`` state and is not a reliable measurement. Instead
+ this test directly measures the wall-clock time spent inside the
+ four session-level listeners Continuum attaches to
+ ``sa.orm.session.Session`` — ``before_flush``, ``after_flush``,
+ ``after_commit``, ``after_rollback`` — plus Superset's own
+ baseline / snapshot / retention-prune listeners (attached to
+ ``db.session``). The cumulative listener time per save is the
+ marginal overhead version capture adds over a save with
+ versioning removed entirely, because without these listeners
+ the ORM would not execute any of that code.
+
+ The approach:
+ 1. Wrap each known listener with a timing proxy that adds its
+ wall-clock time to a per-save accumulator.
+ 2. Save the same chart N times, recording each save's
+ accumulator value.
+ 3. Compute p50 / p95 of the per-save overhead.
+
+ This matches the measurement intent of SC-004 (how much does
+ versioning cost per save) without the fragility of toggling
+ Continuum mid-test.
+ """
+ self.login(ADMIN_USERNAME)
+
+ chart = db.session.query(Slice).first()
+ assert chart is not None
+
+ # Per-save accumulator incremented by the wrapped listeners.
+ acc = [0.0]
+
+ def wrap_listener(original: Any) -> Any:
+ def wrapper(*args: Any, **kwargs: Any) -> Any:
+ t0 = time.perf_counter()
+ try:
+ return original(*args, **kwargs)
+ finally:
+ acc[0] += time.perf_counter() - t0
+
+ wrapper.__wrapped__ = original # type: ignore[attr-defined]
+ return wrapper
+
+ # Instrument Continuum's four session listeners by detaching the
+ # bound method, wrapping, and re-attaching under a single-use
+ # listener handle we can cleanly remove on teardown.
+ session_target = sa.orm.session.Session
+ attached: list[tuple[str, Any]] = []
+ for event_name, listener in list(versioning_manager.session_listeners.items()):
+ sa.event.remove(session_target, event_name, listener)
+ wrapped = wrap_listener(listener)
+ sa.event.listen(session_target, event_name, wrapped)
+ attached.append((event_name, wrapped))
Review Comment:
Fixed in 8b19744b85. The performance gates now use full save latency, which includes both Continuum listeners and Superset listeners attached to `db.session`; the wrapped-listener timing is diagnostic only.
##########
superset/versioning/activity/orchestrator.py:
##########
@@ -0,0 +1,434 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Top-level orchestrator + query-param parsing + observability.
+
+This is the public entry point for the activity-view read path. One
+function — :func:`get_activity` — dispatches on the path entity's
+model class to assemble the cross-entity activity stream:
+
+1. ``resolve_path_entity`` (queries.py) — resolve UUID → live entity.
+2. ``resolve_scope`` (scope.py) — build the related-entity window list.
+3. ``fetch_change_records`` (queries.py) — pull rows from
+ ``version_changes`` joined with ``version_transaction`` and ``ab_user``.
+4. ``filter_records_by_visibility`` (visibility.py) — silent
+ drop of records the requester can't read.
+5. ``apply_entity_name_denormalization`` (queries.py) — resolve entity names
+ from the shadow row valid at each record's transaction_id.
+6. ``apply_record_decoration`` (render.py) — synthesize the ActivityRecord
+ DTO fields and strip internal-only columns.
+7. Paginate in Python over the post-filter list.
+
+Parameter parsing for the REST endpoints lives here too —
+:func:`parse_activity_query_params` is called by the three
+``/activity/`` endpoint handlers before they call ``get_activity``.
+Same for the observability instrumentation: ``_phase_timer`` and
+``_emit_request_shape_attributes`` emit the per-phase timing and
+request-shape metrics, on the same prefix the cross-coupling test pins.
+"""
+
+from __future__ import annotations
+
+import contextlib
+from collections.abc import Iterator
+from datetime import datetime, timezone
+from typing import Any
+from uuid import UUID
+
+from flask import Response
+from flask_appbuilder import Model
+
+from superset.utils import json
+from superset.versioning.activity.kinds import EntityWindows
+from superset.versioning.activity.queries import (
+ apply_entity_name_denormalization,
+ fetch_change_records,
+ first_tracked_tx,
+ mark_first_tracked_saves,
+ resolve_path_entity,
+)
+from superset.versioning.activity.render import apply_record_decoration
+from superset.versioning.activity.scope import resolve_scope
+from superset.versioning.activity.visibility import filter_records_by_visibility
+from superset.versioning.api_helpers import (
+ PathEntityResponseError,
+ resolve_endpoint_path_entity,
+)
+
+_DEFAULT_PAGE_SIZE = 25
+_MAX_PAGE_SIZE = 200
+_VALID_INCLUDE_VALUES: frozenset[str] = frozenset({"self", "related", "all"})
+
+
+# Upper bound on the ``q`` search string. The search is a substring scan over
+# the (already-capped) materialized record set, so this is a cheap-DoS guard,
+# not a correctness limit.
+_MAX_Q_LENGTH = 1024
+
+
+class ActivityParamsError(ValueError):
+ """Raised by :func:`parse_activity_query_params` when a query param is
+ malformed. The endpoint catches this and maps to ``response_400``;
+ no other callers should depend on the exception type."""
+
+
+def parse_activity_query_params(args: Any) -> dict[str, Any]:
+ """Parse the ``since`` / ``until`` / ``include`` / ``page`` / ``page_size``
+ query parameters into the kwargs ``get_activity`` accepts.
+
+ Raises :class:`ActivityParamsError` (subclass of ``ValueError``) when
+ a parameter is malformed. Shared across the three endpoint families
+ (dashboards, charts, datasets) so the parsing and 400-messaging stay
+ consistent.
+ """
+ params: dict[str, Any] = {
+ "include": _parse_include(args.get("include", "all")),
+ "page": _parse_page(args.get("page", "0")),
+ "page_size": _parse_page_size(args.get("page_size")),
+ }
+ if (since := _parse_optional_iso(args.get("since"), name="since")) is not None:
+ params["since"] = since
+ if (until := _parse_optional_iso(args.get("until"), name="until")) is not None:
+ params["until"] = until
+ if q := (args.get("q") or "").strip():
+ if len(q) > _MAX_Q_LENGTH:
+ raise ActivityParamsError(f"'q' must be at most {_MAX_Q_LENGTH} characters")
+ params["q"] = q
+ return params
+
+
+def _parse_optional_iso(raw: str | None, *, name: str) -> datetime | None:
+ """Parse a missing-or-ISO-datetime field; ``None`` for missing,
+ ``ActivityParamsError`` for malformed."""
+ if not raw:
+ return None
+ parsed = _parse_iso_datetime(raw)
+ if parsed is None:
+ raise ActivityParamsError(f"Invalid {name!r} datetime: {raw!r}")
+ return parsed
+
+
+def _parse_include(value: str) -> str:
+ if value not in _VALID_INCLUDE_VALUES:
+ raise ActivityParamsError(
+ f"Invalid 'include' value: {value!r}; "
+ f"must be one of {sorted(_VALID_INCLUDE_VALUES)}"
+ )
+ return value
+
+
+def _parse_page(raw: str) -> int:
+ try:
+ value = int(raw)
+ except (TypeError, ValueError) as exc:
+ raise ActivityParamsError(f"Invalid 'page' value: {raw!r}") from exc
+ if value < 0:
+ raise ActivityParamsError("Invalid 'page' value: must be >= 0")
+ return value
+
+
+def _parse_page_size(raw: str | None) -> int:
+ """``page_size`` honours the default when missing, raises when invalid,
+ and silently clamps to ``_MAX_PAGE_SIZE`` (so ``?page_size=500``
+ returns 200 records instead of a 400)."""
+ if raw is None:
+ return _DEFAULT_PAGE_SIZE
+ try:
+ value = int(raw)
+ except (TypeError, ValueError) as exc:
+ raise ActivityParamsError(f"Invalid 'page_size' value: {raw!r}") from exc
+ if value < 1:
+ raise ActivityParamsError("Invalid 'page_size' value: must be >= 1")
+ return min(value, _MAX_PAGE_SIZE)
+
+
+def _parse_iso_datetime(value: str) -> datetime | None:
+ """Parse an ISO-8601 datetime string. Tolerates the trailing ``Z``
+ suffix that Python <3.11 ``fromisoformat`` rejects, and normalises any
+ timezone-aware result to naive UTC.
+
+ The ``since`` / ``until`` filters bind directly against
+ ``version_transaction.issued_at``, which is ``sa.DateTime()`` — a
+ timezone-*naive* column (UTC by convention). Binding a tz-aware value
+ against it shifts the comparison by the session offset on PostgreSQL
+ (and raises on some drivers), so collapse aware inputs to naive UTC
+ here. Naive inputs pass through unchanged (already treated as UTC).
+ """
+ candidate = value[:-1] + "+00:00" if value.endswith("Z") else value
+ try:
+ parsed = datetime.fromisoformat(candidate)
+ except ValueError:
+ return None
+ if parsed.tzinfo is not None:
+ parsed = parsed.astimezone(timezone.utc).replace(tzinfo=None)
+ return parsed
+
+
+def _record_matches(record: dict[str, Any], q: str) -> bool:
+ """Case-insensitive substring match for the ``q`` search filter,
+ over the human-meaningful surfaces of a decorated activity record:
+ ``summary``, ``entity_name``, ``kind``, the joined ``path`` segments,
+ and the JSON form of ``from_value`` / ``to_value`` (JSON, not Python
+ ``str()``: the client searches the serialized text it renders, so
+ ``false`` / ``null`` / double-quoted keys must match — and falsy
+ values like ``False`` / ``0`` must not collapse to unsearchable
+ empty strings).
+ """
+
+ def _value_text(value: Any) -> str:
+ if value is None:
+ return ""
+ try:
+ return json.dumps(value)
+ except (TypeError, ValueError):
+ return str(value)
+
+ needle = q.lower()
+ haystacks = (
+ record.get("summary") or "",
+ record.get("entity_name") or "",
+ record.get("kind") or "",
+ " ".join(str(seg) for seg in (record.get("path") or [])),
+ _value_text(record.get("from_value")),
+ _value_text(record.get("to_value")),
+ )
+ return any(needle in h.lower() for h in haystacks)
+
+
+def get_activity(
+ model_cls: type[Model],
+ entity_uuid: UUID,
+ *,
+ since: datetime | None = None,
+ until: datetime | None = None,
+ include: str = "all",
+ q: str | None = None,
+ page: int = 0,
+ page_size: int = _DEFAULT_PAGE_SIZE,
+ resolved_entity: Any | None = None,
+) -> tuple[list[dict[str, Any]], int, bool]:
+ """Cross-entity activity stream for one path entity.
+
+ Single polymorphic entry point. Dispatches on *model_cls* to
+ assemble the path entity's self records plus the transitive related-
+ entity records (charts attached to a dashboard, datasets a chart
+ pointed at, etc.).
+
+ Returns ``(records, total_count, truncated)``. ``truncated`` is
+ ``True`` when the per-kind fetch ceiling
+ (``queries._MAX_FETCHED_RECORDS``) bit for any kind — the stream is
+ clamped to a clean time cut and older records exist beyond it, so
+ ``count`` is a floor, not the absolute total.
+
+ *resolved_entity*, when supplied, is the already-resolved live path
+ entity (the endpoint resolves it once via ``resolve_endpoint_path_entity``
+ for the ``raise_for_access`` gate); passing it here skips a second
+ identical ``find_active_by_uuid`` lookup and the TOCTOU window between
+ them. The count is post-visibility (silent visibility filter),
+ post-include-filter, and — when ``q`` is supplied — post-
+ search-filter (``count`` reflects the matches, the contract the
+ server-side search exists to provide), not just the size of the
+ returned slice — clients paginate by passing ``page`` forward until
+ ``page * page_size >= count``.
+
+ Raises ``DashboardNotFoundError`` / ``ChartNotFoundError`` /
+ ``DatasetNotFoundError`` when the path entity doesn't exist.
+ """
+ if resolved_entity is not None:
+ path_entity, path_id = resolved_entity, resolved_entity.id
+ else:
+ path_entity, path_id = resolve_path_entity(model_cls, entity_uuid)
+ path_kind = model_cls.__name__
+ kind_key = path_kind.lower() # "dashboard" / "slice" / "sqlatable"
+
+ # Lower-bound the self window at the entity's first tracked transaction
+ # (matched on (id, uuid)) so a reused integer id can't inherit a
+ # hard-deleted predecessor's history.
+ self_start_tx = first_tracked_tx(model_cls, path_id, path_entity.uuid)
+ with _phase_timer(kind_key, "relationship_resolution_ms"):
+ entity_windows = resolve_scope(path_kind, path_id, include, self_start_tx)
+ if not entity_windows:
+ _emit_request_shape_attributes(
+ kind_key,
+ include=include,
+ has_since_filter=since is not None,
+ page_size=page_size,
+ record_count=0,
+ entity_windows=[],
+ path_kind=path_kind,
+ path_id=path_id,
+ )
+ return [], 0, False
+
+ # Visibility filter runs before decoration: it needs the raw
+ # ``entity_id`` column (which decoration strips), and dropping
+ # invisible records early means we don't pay for name lookup +
+ # tombstone probes + impact counts on records the requester
+ # can't see (the silent-filter contract).
+ with _phase_timer(kind_key, "fetch_ms"):
+ records, truncated = fetch_change_records(entity_windows, since, until)
+ with _phase_timer(kind_key, "visibility_filter_ms"):
+ records = filter_records_by_visibility(records)
+ with _phase_timer(kind_key, "denormalize_ms"):
+ apply_entity_name_denormalization(records)
+ # Runs post-visibility (fewer entities to probe) and pre-
+ # decoration (needs the raw table-form entity_kind/entity_id
+ # that decoration rewrites).
+ mark_first_tracked_saves(records)
+ with _phase_timer(kind_key, "decorate_ms"):
+ apply_record_decoration(records, path_kind, path_id)
+
+ # Server-side search (the panel's client-side
+ # search only covers loaded pages). Applied post-decoration so the
+ # synthesized ``summary`` / ``entity_name`` participate, and pre-
+ # count so pagination paginates the MATCHES — the full record set
+ # is already materialized in Python (the documented design),
+ # so the filter adds no extra query.
+ if q:
+ records = [r for r in records if _record_matches(r, q)]
+
+ total = len(records)
+ bounded_size = max(1, min(page_size, _MAX_PAGE_SIZE))
+ offset = max(0, page) * bounded_size
+
+ _emit_request_shape_attributes(
+ kind_key,
+ include=include,
+ has_since_filter=since is not None,
+ page_size=bounded_size,
+ record_count=total,
+ entity_windows=entity_windows,
+ path_kind=path_kind,
+ path_id=path_id,
+ )
+
+ return records[offset : offset + bounded_size], total, truncated
+
+
+def activity_endpoint(
+ api: Any, model_cls: type[Model], uuid_str: str, request_args: Any
+) -> Response:
+ """Body of ``GET /api/v1/{resource}/<uuid>/activity/``.
+
+ Same shape as :func:`superset.versioning.api_helpers.list_versions_endpoint`
+ for the ``/versions/`` endpoint family. Resolves the path entity,
+ parses the request query params, runs :func:`get_activity`, and
+ wraps the result through ``ActivityResponseSchema``.
+
+ *api* is the FAB ``ModelRestApi`` instance (pass ``self`` from the
+ endpoint method). *request_args* is ``request.args`` from
+ ``flask.request`` — passed explicitly so the helper is testable
+ without a live Flask context.
+ """
+ # pylint: disable=import-outside-toplevel
+ from superset.versioning.schemas import ActivityResponseSchema
+
+ try:
+ entity, _ = resolve_endpoint_path_entity(api, model_cls, uuid_str)
+ except PathEntityResponseError as exc:
+ return exc.response
+
+ try:
+ params = parse_activity_query_params(request_args)
+ except ActivityParamsError as exc:
+ return api.response_400(message=str(exc))
+
+ records, count, truncated = get_activity(
+ model_cls, entity.uuid, resolved_entity=entity, **params
+ )
+ payload = ActivityResponseSchema().dump(
+ {"result": records, "count": count, "truncated": truncated}
+ )
+ return api.response(200, **payload)
+
+
+# ---- Observability -------------------------------------------------------
+
+#: Common prefix for every metric this module emits.
+_METRIC_PREFIX = "superset.activity_view"
+
+
+@contextlib.contextmanager
+def _phase_timer(kind_key: str, phase: str) -> Iterator[None]:
+ """Time the wrapped block and emit
+ ``superset.activity_view.<kind>.<phase>`` to ``stats_logger_manager``.
+ Wrapper around :func:`superset.utils.decorators.stats_timing` that
+ centralises the key construction.
+ """
+ # pylint: disable=import-outside-toplevel
+ from superset.extensions import stats_logger_manager
+ from superset.utils.decorators import stats_timing
+
+ with stats_timing(
+ f"{_METRIC_PREFIX}.{kind_key}.{phase}",
+ stats_logger_manager.instance,
+ ):
+ yield
+
+
+def _emit_request_shape_attributes(
+ kind_key: str,
+ *,
+ include: str,
+ has_since_filter: bool,
+ page_size: int,
+ record_count: int,
+ entity_windows: list[EntityWindows],
+ path_kind: str,
+ path_id: int,
+) -> None:
+ """Emit non-PII shape counters about the request and its result set.
+
+ Emits: include_mode / has_since_filter / page_size / record_count
+ + per-related-kind entity counts. **No PII**: entity names, diff
+ content, user identifiers — none of those reach the metric layer.
+ The counters use ``incr`` (counters) since they're tags, not
+ latencies; the timing keys above carry the latency dimension.
+ """
+ # pylint: disable=import-outside-toplevel
+ from superset.extensions import stats_logger_manager
+
+ sl = stats_logger_manager.instance
+
+ # Tag-style metrics: one counter per attribute value. The statsd
+ # bridge accepts arbitrary strings; downstream dashboards filter by
+ # the value segment.
+ sl.incr(f"{_METRIC_PREFIX}.{kind_key}.requests.include_{include}")
+ sl.incr(
+ f"{_METRIC_PREFIX}.{kind_key}.requests."
+ f"has_since_filter_{'true' if has_since_filter else 'false'}"
+ )
+ sl.gauge(f"{_METRIC_PREFIX}.{kind_key}.page_size", float(page_size))
+ sl.gauge(f"{_METRIC_PREFIX}.{kind_key}.record_count", float(record_count))
+
+ # Per-related-kind entity counts. The scope
+ # list includes the path entity itself (the "self" window); exclude
+ # it so the gauge reflects only the *related* entities the request
+ # fanned out to, not "this request touched itself".
+ by_kind: dict[str, int] = {"Slice": 0, "SqlaTable": 0, "Dashboard": 0}
+ for api_kind, entity_id, _windows in entity_windows:
+ if (api_kind, entity_id) == (path_kind, path_id):
+ continue
+ if api_kind in by_kind:
+ by_kind[api_kind] += 1
+ sl.gauge(
+ f"{_METRIC_PREFIX}.{kind_key}.related_entity_count.charts",
+ float(by_kind["Slice"]),
+ )
+ sl.gauge(
+ f"{_METRIC_PREFIX}.{kind_key}.related_entity_count.datasets",
+ float(by_kind["SqlaTable"]),
Review Comment:
Addressed by subsequent commits; the flagged code is no longer current after the rebase onto master. Resolving as outdated.
##########
superset/dashboards/api.py:
##########
@@ -2540,6 +2539,85 @@ def get_version(self, uuid_str: str, version_uuid_str: str) -> Response:
404:
$ref: '#/components/responses/404'
"""
- return get_version_endpoint(
- self, Dashboard, uuid_str, version_uuid_str, access_kwarg="dashboard"
- )
+ return get_version_endpoint(self, Dashboard, uuid_str, version_uuid_str)
+
+ @expose("/<uuid_str>/activity/", methods=("GET",))
+ @protect()
+ @safe
+ @statsd_metrics
+ @event_logger.log_this_with_context(
+ action=lambda self, *args, **kwargs: f"{self.__class__.__name__}.activity",
+ log_to_statsd=False,
+ )
+ def activity(self, uuid_str: str) -> Response:
Review Comment:
Fixed in 8b19744b85 by mapping the activity endpoint to the standard `get` permission.
##########
superset/datasets/api.py:
##########
@@ -1808,6 +1807,86 @@ def get_version(self, uuid_str: str, version_uuid_str: str) -> Response:
404:
$ref: '#/components/responses/404'
"""
- return get_version_endpoint(
- self, SqlaTable, uuid_str, version_uuid_str, access_kwarg="datasource"
- )
+ return get_version_endpoint(self, SqlaTable, uuid_str, version_uuid_str)
+
+ @expose("/<uuid_str>/activity/", methods=("GET",))
+ @protect()
+ @safe
+ @statsd_metrics
+ @event_logger.log_this_with_context(
+ action=lambda self, *args, **kwargs: f"{self.__class__.__name__}.activity",
+ log_to_statsd=False,
+ )
+ def activity(self, uuid_str: str) -> Response:
Review Comment:
Fixed in 8b19744b85 by mapping the activity endpoint to the standard `get` permission.
##########
superset/versioning/activity/queries.py:
##########
@@ -0,0 +1,754 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""DB-touching helpers for the activity-view read path.
+
+All Phase A relationship walks (``charts_attached_to_dashboard``,
+``datasets_used_by_chart``, ``batch_datasets_used_by_charts``),
+the Phase B change-record fetch (``fetch_change_records`` /
+``_select_change_rows_for_kinds``), the name-denormalization helpers
+(``_resolve_names_for_kind`` / ``apply_entity_name_denormalization``), the
+path-entity resolution helper (``resolve_path_entity``), and the
+tombstone-state lookup (``check_entity_tombstones``) live here.
+
+Each helper is a thin SELECT-and-shape function — no orchestration,
+no business logic. Callers in :mod:`scope`, :mod:`render`, and
+:mod:`orchestrator` compose them into the end-to-end request.
+
+**Inline imports.** Continuum's ``version_class`` / ``versioning_manager``
+and the Superset model classes are imported inside each helper because
+this package is loaded from ``init_versioning()`` before all SQLAlchemy
+mappers are configured.
+"""
+
+from __future__ import annotations
+
+import logging
+from datetime import datetime
+from typing import Any
+from uuid import UUID
+
+import sqlalchemy as sa
+from flask_appbuilder import Model
+
+from superset.extensions import db
+from superset.versioning.activity.kinds import (
+ API_KIND_TO_TABLE,
+ chunked_ids,
+ EntityWindows,
+ load_live_model,
+ NAME_COLUMN,
+ NOT_FOUND_EXC,
+ TABLE_KIND_TO_API,
+ Window,
+)
+from superset.versioning.activity.windows import row_within_any_window
+from superset.versioning.changes import version_changes_table
+
+logger = logging.getLogger(__name__)
+
+# ---- Path-entity resolution -----------------------------------------------
+
+
+def resolve_path_entity(model_cls: type[Model], entity_uuid: UUID) -> tuple[Any, int]:
+ """Resolve *entity_uuid* to ``(live_entity, entity_id)`` or raise a
+ typed 404.
+
+ Soft-delete handling is inherited transparently from
+ :func:`superset.versioning.queries.find_active_by_uuid` once it
+ learns to filter out ``deleted_at IS NOT NULL`` rows; at that point
+ soft-deleted paths will also raise here.
+ """
+ # pylint: disable=import-outside-toplevel
+ from superset.versioning.queries import find_active_by_uuid
+
+ entity = find_active_by_uuid(model_cls, entity_uuid)
+ if entity is None:
+ api_kind = model_cls.__name__
+ exc_cls = NOT_FOUND_EXC.get(api_kind)
+ if exc_cls is None:
+ raise LookupError(
+ f"Activity view does not support model class {api_kind!r}"
+ )
+ raise exc_cls(str(entity_uuid))
+ return entity, entity.id
+
+
+def first_tracked_tx(
+ model_cls: type[Model], entity_id: int, entity_uuid: UUID
+) -> int | None:
+ """Return the earliest Continuum transaction_id of the shadow rows
+ that belong to *this* live entity, matched on ``(id, uuid)``; ``None``
+ when the entity has no tracked history yet.
+
+ Used to lower-bound the self-scope window. Matching on the integer
+ ``id`` alone would inherit a previously hard-deleted entity's history
+ under id reuse (SQLite/MySQL reuse ``max(id)+1``); pinning the uuid
+ too — the same discrimination :func:`mark_first_tracked_saves` uses —
+ scopes the window to the current entity's own transactions.
+ """
+ # pylint: disable=import-outside-toplevel
+ from sqlalchemy_continuum import version_class
+
+ shadow_tbl = version_class(model_cls).__table__
+ stmt = sa.select(sa.func.min(shadow_tbl.c.transaction_id)).where(
+ shadow_tbl.c.id == entity_id,
+ shadow_tbl.c.uuid == entity_uuid,
+ )
+ return db.session.connection().execute(stmt).scalar()
+
+
+# ---- Phase A: relationship-traversal queries ------------------------------
+
+
+def charts_attached_to_dashboard(dashboard_id: int) -> list[tuple[int, Window]]:
+ """Return ``(slice_id, window)`` for every chart that has ever been on
+ *dashboard_id*, with each association's validity window in
+ transaction-id space.
+
+ Reads from ``dashboard_slices_version`` (Continuum's auto-generated
+ M2M shadow). Rows with ``operation_type = 2`` (DELETE) are excluded
+ so we don't synthesize a phantom window from a detachment row.
+ """
+ # pylint: disable=import-outside-toplevel
+ from sqlalchemy_continuum import version_class
+
+ from superset.models.dashboard import Dashboard
+
+ metadata = version_class(Dashboard).__table__.metadata
+ m2m_tbl = metadata.tables.get("dashboard_slices_version")
+ if m2m_tbl is None:
+ return []
+
+ rows = (
+ db.session.connection()
+ .execute(
+ sa.select(
+ m2m_tbl.c.slice_id,
+ m2m_tbl.c.transaction_id,
+ m2m_tbl.c.end_transaction_id,
+ ).where(
+ m2m_tbl.c.dashboard_id == dashboard_id,
+ m2m_tbl.c.operation_type != 2,
+ m2m_tbl.c.slice_id.is_not(None),
+ )
+ )
+ .all()
+ )
+ result: list[tuple[int, Window]] = []
+ for row in rows:
+ try:
+ window = Window(row[1], row[2])
+ except ValueError:
+ # A degenerate shadow row (end_tx <= start_tx) must not 500 the
+ # endpoint; skip it and leave a breadcrumb for investigation.
+ logger.warning(
+ "activity: skipping degenerate dashboard_slices_version row "
+ "(dashboard_id=%s, slice_id=%s, tx=%s, end_tx=%s)",
+ dashboard_id,
+ row[0],
+ row[1],
+ row[2],
+ )
+ continue
+ result.append((row[0], window))
+ return result
+
+
+def datasets_used_by_chart(slice_id: int) -> list[tuple[int, Window]]:
+ """Return ``(datasource_id, window)`` for every dataset that *slice_id*
+ has ever pointed at, with each association's validity window.
+
+ Single-slice form, used by ``_resolve_chart_scope`` where there
+ is only one chart to walk. The dashboard-scope path calls
+ :func:`batch_datasets_used_by_charts` instead so the query fires
+ once for all slices on the dashboard, not once per slice.
+
+ Reads from ``slices_version`` (the chart parent shadow). Filters to
+ ``datasource_type = 'table'`` because the activity view only follows
+ the chart → ``SqlaTable`` dependency edge (not legacy/other
+ datasources). Rows with ``operation_type = 2`` are excluded.
+ """
+ return batch_datasets_used_by_charts({slice_id}).get(slice_id, [])
+
+
+def batch_datasets_used_by_charts(
+ slice_ids: set[int],
+) -> dict[int, list[tuple[int, Window]]]:
+ """Batch form of :func:`datasets_used_by_chart`. Returns
+ ``{slice_id: [(dataset_id, window), ...]}`` in a single query so the
+ dashboard-scope walker doesn't fire one query per chart on the
+ dashboard. The previous per-slice shape became O(n_charts) round-
+ trips, which dominated ``get_activity`` latency on dashboards with
+ rich history (profile run 2026-05-26 showed `resolve_scope`
+ accounting for ~1.9s out of 4s p95).
+ """
+ if not slice_ids:
+ return {}
+
+ # pylint: disable=import-outside-toplevel
+ from sqlalchemy_continuum import version_class
+
+ from superset.models.slice import Slice
+
+ slices_tbl = version_class(Slice).__table__
+ grouped: dict[int, list[tuple[int, Window]]] = {}
+ # Chunk the IN-clause under SQLite's bind-variable floor (a dashboard can
+ # carry more charts than the floor allows in one statement).
+ rows: list[Any] = []
+ for chunk in chunked_ids(slice_ids):
+ rows.extend(
+ db.session.connection()
+ .execute(
+ sa.select(
+ slices_tbl.c.id,
+ slices_tbl.c.datasource_id,
+ slices_tbl.c.transaction_id,
+ slices_tbl.c.end_transaction_id,
+ ).where(
+ slices_tbl.c.id.in_(chunk),
+ slices_tbl.c.datasource_type == "table",
+ slices_tbl.c.operation_type != 2,
+ slices_tbl.c.datasource_id.is_not(None),
+ )
+ )
+ .mappings()
+ .all()
+ )
+ for row in rows:
+ try:
+ window = Window(row["transaction_id"], row["end_transaction_id"])
+ except ValueError:
+ # A degenerate shadow row (end_tx <= start_tx) must not 500 the
+ # endpoint; skip it and leave a breadcrumb for investigation.
+ logger.warning(
+ "activity: skipping degenerate slices_version row "
+ "(slice_id=%s, datasource_id=%s, tx=%s, end_tx=%s)",
+ row["id"],
+ row["datasource_id"],
+ row["transaction_id"],
+ row["end_transaction_id"],
+ )
+ continue
+ grouped.setdefault(row["id"], []).append((row["datasource_id"], window))
+ return grouped
+
+
+# ---- Phase B: change-record fetch -----------------------------------------
+
+
+def fetch_change_records(
+ entity_window_tuples: list[EntityWindows],
+ since: datetime | None,
+ until: datetime | None,
+) -> tuple[list[dict[str, Any]], bool]:
+ """Fetch all ``version_changes`` rows matching any of the supplied
+ entity-window tuples, joined with ``version_transaction`` for
+ ``issued_at`` and ``user_id``.
+
+ Each tuple is ``(api_kind, entity_id, [(start_tx, end_tx), ...])``;
+ a record matches when ``entity_kind`` equals the table-stored form
+ of *api_kind*, ``entity_id`` matches, and ``transaction_id`` falls
+ inside at least one of the entity's windows. ``since``/``until``
+ further restrict by ``issued_at``.
+
+ Implementation: one SELECT per kind with ``entity_id IN (...)`` and
+ a wide ``transaction_id`` bound (the union of all windows for that
+ kind). Per-window precision is applied in Python afterward. This
+ keeps the SQL shape proportional to the number of *kinds* (≤3) and
+ the bound proportional to the union of windows, not the cross-
+ product of (entity, window) — which previously generated one OR
+ clause per (entity, window) pair and hit SQLite's
+ ``SQLITE_MAX_EXPR_DEPTH`` limit on dashboards with many slices
+ or many historical attachment windows.
+
+ The visibility filter runs after this function (records
+ the requester can't read are silently dropped and must not
+ contribute to ``count``), so the orchestrator paginates in Python
+ over the filtered list — there is no DB-level page ``OFFSET`` here.
+ There IS a per-kind safety ceiling (``_MAX_FETCHED_RECORDS``) that
+ bounds how much history a single request materializes (to at most
+ ``n_kinds * _MAX_FETCHED_RECORDS``); when a kind exhausts it, the
+ second return value is ``True``, the returned stream is clamped to a
+ clean time cut (records older than the highest per-kind fetch floor
+ are dropped so truncation is a time boundary, not per-kind holes), and
+ the caller surfaces ``truncated`` on the response.
+
+ Returns ``(records, truncated)``. Records are ordered by
+ ``(issued_at DESC, transaction_id DESC, sequence DESC)`` — the
+ secondary keys break ties for the stable-ordering contract.
+ """
+ if not entity_window_tuples:
+ return [], False
+
+ # Group windows by (table_kind, entity_id) and by table_kind for SQL
+ # narrowing. The fetch is per-kind; the post-filter is per-entity.
+ windows_by_entity: dict[tuple[str, int], list[Window]] = {}
+ ids_by_kind: dict[str, set[int]] = {}
+ for api_kind, entity_id, windows in entity_window_tuples:
+ table_kind = API_KIND_TO_TABLE.get(api_kind)
+ if table_kind is None or not windows:
+ continue
+ ids_by_kind.setdefault(table_kind, set()).add(entity_id)
+ windows_by_entity.setdefault((table_kind, entity_id), []).extend(windows)
+
+ if not ids_by_kind:
+ return [], False
+
+ # Per-kind transaction_id bounds = the union of that kind's windows.
+ # Pushing these into the SQL WHERE ensures the per-statement LIMIT
+ # selects from IN-WINDOW rows. Without it, a related entity whose
+ # association window is far in the past would have the newest ``limit``
+ # (out-of-window) rows fetched and discarded, silently dropping its
+ # in-window records that lie beyond the limit. ``end_tx = None``
+ # (open-ended/current) means no upper bound for that kind.
+ bounds_by_kind: dict[str, tuple[int, int | None]] = {}
+ for (table_kind, _entity_id), windows in windows_by_entity.items():
+ for w in windows:
+ cur = bounds_by_kind.get(table_kind)
+ if cur is None:
+ bounds_by_kind[table_kind] = (w.start_tx, w.end_tx)
+ continue
+ lo = min(cur[0], w.start_tx)
+ hi = None if (cur[1] is None or w.end_tx is None) else max(cur[1], w.end_tx)
+ bounds_by_kind[table_kind] = (lo, hi)
+
+ rows, truncated, truncation_floor = _select_change_rows_for_kinds(
+ ids_by_kind, bounds_by_kind, since, until, _MAX_FETCHED_RECORDS
+ )
+ filtered = [
+ row
+ for row in rows
+ if row_within_any_window(
+ row, windows_by_entity.get((row["entity_kind"], row["entity_id"]), [])
+ )
+ ]
+ if truncated and truncation_floor is not None:
+ # Collapse truncation to a clean time cut. Below the highest per-kind
+ # fetch floor at least one kind is missing rows, so surfacing the
+ # other kinds there would present an incomplete ("nothing else
+ # changed") picture. Drop everything older than the floor so
+ # ``truncated=True`` means "complete at/after this instant, nothing
+ # shown before it" — a time boundary rather than per-kind holes.
+ filtered = [r for r in filtered if r["issued_at"] >= truncation_floor]
Review Comment:
Addressed by subsequent commits; the flagged code is no longer current after the rebase onto master. Resolving as outdated.
##########
superset/versioning/activity/queries.py:
##########
@@ -0,0 +1,754 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""DB-touching helpers for the activity-view read path.
+
+All Phase A relationship walks (``charts_attached_to_dashboard``,
+``datasets_used_by_chart``, ``batch_datasets_used_by_charts``),
+the Phase B change-record fetch (``fetch_change_records`` /
+``_select_change_rows_for_kinds``), the name-denormalization helpers
+(``_resolve_names_for_kind`` / ``apply_entity_name_denormalization``), the
+path-entity resolution helper (``resolve_path_entity``), and the
+tombstone-state lookup (``check_entity_tombstones``) live here.
+
+Each helper is a thin SELECT-and-shape function — no orchestration,
+no business logic. Callers in :mod:`scope`, :mod:`render`, and
+:mod:`orchestrator` compose them into the end-to-end request.
+
+**Inline imports.** Continuum's ``version_class`` / ``versioning_manager``
+and the Superset model classes are imported inside each helper because
+this package is loaded from ``init_versioning()`` before all SQLAlchemy
+mappers are configured.
+"""
+
+from __future__ import annotations
+
+import logging
+from datetime import datetime
+from typing import Any
+from uuid import UUID
+
+import sqlalchemy as sa
+from flask_appbuilder import Model
+
+from superset.extensions import db
+from superset.versioning.activity.kinds import (
+ API_KIND_TO_TABLE,
+ chunked_ids,
+ EntityWindows,
+ load_live_model,
+ NAME_COLUMN,
+ NOT_FOUND_EXC,
+ TABLE_KIND_TO_API,
+ Window,
+)
+from superset.versioning.activity.windows import row_within_any_window
+from superset.versioning.changes import version_changes_table
+
+logger = logging.getLogger(__name__)
+
+# ---- Path-entity resolution -----------------------------------------------
+
+
+def resolve_path_entity(model_cls: type[Model], entity_uuid: UUID) -> tuple[Any, int]:
+ """Resolve *entity_uuid* to ``(live_entity, entity_id)`` or raise a
+ typed 404.
+
+ Soft-delete handling is inherited transparently from
+ :func:`superset.versioning.queries.find_active_by_uuid` once it
+ learns to filter out ``deleted_at IS NOT NULL`` rows; at that point
+ soft-deleted paths will also raise here.
+ """
+ # pylint: disable=import-outside-toplevel
+ from superset.versioning.queries import find_active_by_uuid
+
+ entity = find_active_by_uuid(model_cls, entity_uuid)
+ if entity is None:
+ api_kind = model_cls.__name__
+ exc_cls = NOT_FOUND_EXC.get(api_kind)
+ if exc_cls is None:
+ raise LookupError(
+ f"Activity view does not support model class {api_kind!r}"
+ )
+ raise exc_cls(str(entity_uuid))
+ return entity, entity.id
+
+
+def first_tracked_tx(
+ model_cls: type[Model], entity_id: int, entity_uuid: UUID
+) -> int | None:
+ """Return the earliest Continuum transaction_id of the shadow rows
+ that belong to *this* live entity, matched on ``(id, uuid)``; ``None``
+ when the entity has no tracked history yet.
+
+ Used to lower-bound the self-scope window. Matching on the integer
+ ``id`` alone would inherit a previously hard-deleted entity's history
+ under id reuse (SQLite/MySQL reuse ``max(id)+1``); pinning the uuid
+ too — the same discrimination :func:`mark_first_tracked_saves` uses —
+ scopes the window to the current entity's own transactions.
+ """
+ # pylint: disable=import-outside-toplevel
+ from sqlalchemy_continuum import version_class
+
+ shadow_tbl = version_class(model_cls).__table__
+ stmt = sa.select(sa.func.min(shadow_tbl.c.transaction_id)).where(
+ shadow_tbl.c.id == entity_id,
+ shadow_tbl.c.uuid == entity_uuid,
+ )
+ return db.session.connection().execute(stmt).scalar()
+
+
+# ---- Phase A: relationship-traversal queries ------------------------------
+
+
+def charts_attached_to_dashboard(dashboard_id: int) -> list[tuple[int, Window]]:
+ """Return ``(slice_id, window)`` for every chart that has ever been on
+ *dashboard_id*, with each association's validity window in
+ transaction-id space.
+
+ Reads from ``dashboard_slices_version`` (Continuum's auto-generated
+ M2M shadow). Rows with ``operation_type = 2`` (DELETE) are excluded
+ so we don't synthesize a phantom window from a detachment row.
+ """
+ # pylint: disable=import-outside-toplevel
+ from sqlalchemy_continuum import version_class
+
+ from superset.models.dashboard import Dashboard
+
+ metadata = version_class(Dashboard).__table__.metadata
+ m2m_tbl = metadata.tables.get("dashboard_slices_version")
+ if m2m_tbl is None:
+ return []
+
+ rows = (
+ db.session.connection()
+ .execute(
+ sa.select(
+ m2m_tbl.c.slice_id,
+ m2m_tbl.c.transaction_id,
+ m2m_tbl.c.end_transaction_id,
+ ).where(
+ m2m_tbl.c.dashboard_id == dashboard_id,
+ m2m_tbl.c.operation_type != 2,
+ m2m_tbl.c.slice_id.is_not(None),
+ )
+ )
+ .all()
+ )
+ result: list[tuple[int, Window]] = []
+ for row in rows:
+ try:
+ window = Window(row[1], row[2])
+ except ValueError:
+ # A degenerate shadow row (end_tx <= start_tx) must not 500 the
+ # endpoint; skip it and leave a breadcrumb for investigation.
+ logger.warning(
+ "activity: skipping degenerate dashboard_slices_version row "
+ "(dashboard_id=%s, slice_id=%s, tx=%s, end_tx=%s)",
+ dashboard_id,
+ row[0],
+ row[1],
+ row[2],
+ )
+ continue
+ result.append((row[0], window))
+ return result
+
+
+def datasets_used_by_chart(slice_id: int) -> list[tuple[int, Window]]:
+ """Return ``(datasource_id, window)`` for every dataset that *slice_id*
+ has ever pointed at, with each association's validity window.
+
+ Single-slice form, used by ``_resolve_chart_scope`` where there
+ is only one chart to walk. The dashboard-scope path calls
+ :func:`batch_datasets_used_by_charts` instead so the query fires
+ once for all slices on the dashboard, not once per slice.
+
+ Reads from ``slices_version`` (the chart parent shadow). Filters to
+ ``datasource_type = 'table'`` because the activity view only follows
+ the chart → ``SqlaTable`` dependency edge (not legacy/other
+ datasources). Rows with ``operation_type = 2`` are excluded.
+ """
+ return batch_datasets_used_by_charts({slice_id}).get(slice_id, [])
+
+
+def batch_datasets_used_by_charts(
+ slice_ids: set[int],
+) -> dict[int, list[tuple[int, Window]]]:
+ """Batch form of :func:`datasets_used_by_chart`. Returns
+ ``{slice_id: [(dataset_id, window), ...]}`` in a single query so the
+ dashboard-scope walker doesn't fire one query per chart on the
+ dashboard. The previous per-slice shape became O(n_charts) round-
+ trips, which dominated ``get_activity`` latency on dashboards with
+ rich history (profile run 2026-05-26 showed `resolve_scope`
+ accounting for ~1.9s out of 4s p95).
+ """
+ if not slice_ids:
+ return {}
+
+ # pylint: disable=import-outside-toplevel
+ from sqlalchemy_continuum import version_class
+
+ from superset.models.slice import Slice
+
+ slices_tbl = version_class(Slice).__table__
+ grouped: dict[int, list[tuple[int, Window]]] = {}
+ # Chunk the IN-clause under SQLite's bind-variable floor (a dashboard can
+ # carry more charts than the floor allows in one statement).
+ rows: list[Any] = []
+ for chunk in chunked_ids(slice_ids):
+ rows.extend(
+ db.session.connection()
+ .execute(
+ sa.select(
+ slices_tbl.c.id,
+ slices_tbl.c.datasource_id,
+ slices_tbl.c.transaction_id,
+ slices_tbl.c.end_transaction_id,
+ ).where(
+ slices_tbl.c.id.in_(chunk),
+ slices_tbl.c.datasource_type == "table",
+ slices_tbl.c.operation_type != 2,
+ slices_tbl.c.datasource_id.is_not(None),
+ )
+ )
+ .mappings()
+ .all()
+ )
+ for row in rows:
+ try:
+ window = Window(row["transaction_id"], row["end_transaction_id"])
+ except ValueError:
+ # A degenerate shadow row (end_tx <= start_tx) must not 500 the
+ # endpoint; skip it and leave a breadcrumb for investigation.
+ logger.warning(
+ "activity: skipping degenerate slices_version row "
+ "(slice_id=%s, datasource_id=%s, tx=%s, end_tx=%s)",
+ row["id"],
+ row["datasource_id"],
+ row["transaction_id"],
+ row["end_transaction_id"],
+ )
+ continue
+ grouped.setdefault(row["id"], []).append((row["datasource_id"], window))
+ return grouped
+
+
+# ---- Phase B: change-record fetch -----------------------------------------
+
+
+def fetch_change_records(
+ entity_window_tuples: list[EntityWindows],
+ since: datetime | None,
+ until: datetime | None,
+) -> tuple[list[dict[str, Any]], bool]:
+ """Fetch all ``version_changes`` rows matching any of the supplied
+ entity-window tuples, joined with ``version_transaction`` for
+ ``issued_at`` and ``user_id``.
+
+ Each tuple is ``(api_kind, entity_id, [(start_tx, end_tx), ...])``;
+ a record matches when ``entity_kind`` equals the table-stored form
+ of *api_kind*, ``entity_id`` matches, and ``transaction_id`` falls
+ inside at least one of the entity's windows. ``since``/``until``
+ further restrict by ``issued_at``.
+
+ Implementation: one SELECT per kind with ``entity_id IN (...)`` and
+ a wide ``transaction_id`` bound (the union of all windows for that
+ kind). Per-window precision is applied in Python afterward. This
+ keeps the SQL shape proportional to the number of *kinds* (≤3) and
+ the bound proportional to the union of windows, not the cross-
+ product of (entity, window) — which previously generated one OR
+ clause per (entity, window) pair and hit SQLite's
+ ``SQLITE_MAX_EXPR_DEPTH`` limit on dashboards with many slices
+ or many historical attachment windows.
+
+ The visibility filter runs after this function (records
+ the requester can't read are silently dropped and must not
+ contribute to ``count``), so the orchestrator paginates in Python
+ over the filtered list — there is no DB-level page ``OFFSET`` here.
+ There IS a per-kind safety ceiling (``_MAX_FETCHED_RECORDS``) that
+ bounds how much history a single request materializes (to at most
+ ``n_kinds * _MAX_FETCHED_RECORDS``); when a kind exhausts it, the
+ second return value is ``True``, the returned stream is clamped to a
+ clean time cut (records older than the highest per-kind fetch floor
+ are dropped so truncation is a time boundary, not per-kind holes), and
+ the caller surfaces ``truncated`` on the response.
+
+ Returns ``(records, truncated)``. Records are ordered by
+ ``(issued_at DESC, transaction_id DESC, sequence DESC)`` — the
+ secondary keys break ties for the stable-ordering contract.
+ """
+ if not entity_window_tuples:
+ return [], False
+
+ # Group windows by (table_kind, entity_id) and by table_kind for SQL
+ # narrowing. The fetch is per-kind; the post-filter is per-entity.
+ windows_by_entity: dict[tuple[str, int], list[Window]] = {}
+ ids_by_kind: dict[str, set[int]] = {}
+ for api_kind, entity_id, windows in entity_window_tuples:
+ table_kind = API_KIND_TO_TABLE.get(api_kind)
+ if table_kind is None or not windows:
+ continue
+ ids_by_kind.setdefault(table_kind, set()).add(entity_id)
+ windows_by_entity.setdefault((table_kind, entity_id), []).extend(windows)
+
+ if not ids_by_kind:
+ return [], False
+
+ # Per-kind transaction_id bounds = the union of that kind's windows.
+ # Pushing these into the SQL WHERE ensures the per-statement LIMIT
+ # selects from IN-WINDOW rows. Without it, a related entity whose
+ # association window is far in the past would have the newest ``limit``
+ # (out-of-window) rows fetched and discarded, silently dropping its
+ # in-window records that lie beyond the limit. ``end_tx = None``
+ # (open-ended/current) means no upper bound for that kind.
+ bounds_by_kind: dict[str, tuple[int, int | None]] = {}
+ for (table_kind, _entity_id), windows in windows_by_entity.items():
+ for w in windows:
+ cur = bounds_by_kind.get(table_kind)
+ if cur is None:
+ bounds_by_kind[table_kind] = (w.start_tx, w.end_tx)
+ continue
+ lo = min(cur[0], w.start_tx)
+ hi = None if (cur[1] is None or w.end_tx is None) else max(cur[1], w.end_tx)
+ bounds_by_kind[table_kind] = (lo, hi)
+
+ rows, truncated, truncation_floor = _select_change_rows_for_kinds(
+ ids_by_kind, bounds_by_kind, since, until, _MAX_FETCHED_RECORDS
+ )
+ filtered = [
+ row
+ for row in rows
+ if row_within_any_window(
+ row, windows_by_entity.get((row["entity_kind"], row["entity_id"]), [])
+ )
+ ]
+ if truncated and truncation_floor is not None:
+ # Collapse truncation to a clean time cut. Below the highest per-kind
+ # fetch floor at least one kind is missing rows, so surfacing the
+ # other kinds there would present an incomplete ("nothing else
+ # changed") picture. Drop everything older than the floor so
+ # ``truncated=True`` means "complete at/after this instant, nothing
+ # shown before it" — a time boundary rather than per-kind holes.
+ filtered = [r for r in filtered if r["issued_at"] >= truncation_floor]
+ # Sort key must be TOTAL so pagination is stable across requests: two
+ # records from different entities can share (issued_at, transaction_id,
+ # sequence), so append (entity_kind, entity_id) to break remaining ties
+ # deterministically. Without these the relative order of tied records
+ # depends on set-iteration order and a record could shift pages.
+ filtered.sort(
+ key=lambda r: (
+ r["issued_at"],
+ r["transaction_id"],
+ r["sequence"],
+ r["entity_kind"],
+ r["entity_id"],
+ ),
+ reverse=True,
+ )
+ return filtered, truncated
+
+
+def _select_change_rows_for_kinds(
+ ids_by_kind: dict[str, set[int]],
+ bounds_by_kind: dict[str, tuple[int, int | None]],
+ since: datetime | None,
+ until: datetime | None,
+ limit: int,
+) -> tuple[list[dict[str, Any]], bool, datetime | None]:
+ """Fire one SELECT per entity_kind with ``entity_id IN (...)``;
+ concatenate the results. Each SELECT joins ``version_transaction``
+ + ``ab_user`` so the orchestrator has the columns it needs for
+ decoration.
+
+ Per-kind, not one query: SQLAlchemy's ``tuple_(entity_kind,
+ entity_id).in_(...)`` would collapse the three queries into one,
+ but its SQL emission is not portable across Postgres, MySQL, and
+ SQLite. The per-kind shape is the correct trade-off given
+ Superset's multi-dialect requirement (at most 3 round-trips per
+ request, bounded by the kind taxonomy). Do not "optimise" into a
+ composite-tuple IN clause without verifying the SQL on all three
+ dialects.
+
+ **Init-order dependency.** ``tx_tbl.c.action_kind`` resolves only
+ after ``init_versioning()`` has run — the column is appended onto
+ Continuum's transaction Table by
+ ``superset.versioning.factory.VersionTransactionFactory`` at app
+ start via ``append_column`` + ``add_property``. This helper is
+ safe to call from request-path code because the app is fully
+ initialised by then; calling it from a script that imports the
+ versioning package without going through ``init_versioning()``
+ will raise ``AttributeError`` on the ``action_kind`` attribute
+ access below."""
+ # pylint: disable=import-outside-toplevel
+ from sqlalchemy_continuum import versioning_manager
+
+ from superset import security_manager
+
+ tx_tbl = versioning_manager.transaction_cls.__table__
+ user_tbl = security_manager.user_model.__table__
+ vc = version_changes_table
+ join_tree = vc.join(tx_tbl, vc.c.transaction_id == tx_tbl.c.id).outerjoin(
+ user_tbl, tx_tbl.c.user_id == user_tbl.c.id
+ )
+ select_cols = (
+ vc.c.transaction_id,
+ vc.c.entity_kind,
+ vc.c.entity_id,
+ vc.c.sequence,
+ vc.c.kind,
+ vc.c.operation,
+ vc.c.path,
+ vc.c.from_value,
+ vc.c.to_value,
+ tx_tbl.c.issued_at,
+ tx_tbl.c.user_id,
+ # ``action_kind`` is the high-level avenue (restore / import /
+ # clone / NULL=ordinary save) stamped by the originating
+ # command via the change-record listener. All records sharing a
+ # ``transaction_id`` share the same value. The column is
+ # declared on the Continuum Table by ``VersionTransactionFactory``,
+ # so ``tx_tbl.c.action_kind`` resolves cleanly here. See
+ # the three change-record dimensions.
+ tx_tbl.c.action_kind,
+ user_tbl.c.id.label("changed_by_id"),
+ user_tbl.c.first_name,
+ user_tbl.c.last_name,
+ )
+
+ out: list[dict[str, Any]] = []
+ truncated = False
+ truncation_floors: list[datetime] = []
+ for table_kind, entity_ids in ids_by_kind.items():
+ # The fetch budget is per KIND, not per id-chunk. A kind whose
+ # entity_ids span several bind-variable chunks shares one ``limit``
+ # across those chunks, so the rows a single request materializes are
+ # bounded by ``n_kinds * limit`` (<= 3 * _MAX_FETCHED_RECORDS), not
+ # ``n_chunks * limit``. ``entity_ids`` is chunked to stay inside
+ # SQLite's ``SQLITE_MAX_VARIABLE_NUMBER`` floor (default 999 in many
+ # builds); Postgres + MySQL accept the full list but the chunk is
+ # dialect-agnostic for simplicity.
+ kind_rows: list[dict[str, Any]] = []
+ for chunk in chunked_ids(entity_ids):
+ remaining = limit - len(kind_rows)
+ if remaining <= 0:
+ break
+ stmt = (
+ sa.select(*select_cols)
+ .select_from(join_tree)
+ .where(
+ vc.c.entity_kind == table_kind,
+ vc.c.entity_id.in_(chunk),
+ )
+ )
+ # Bound by the kind's window union so the LIMIT picks in-window
+ # rows (see fetch_change_records). The per-entity window filter
+ # still runs in Python afterwards for exact membership.
+ tx_lo, tx_hi = bounds_by_kind[table_kind]
+ stmt = stmt.where(vc.c.transaction_id >= tx_lo)
+ if tx_hi is not None:
+ stmt = stmt.where(vc.c.transaction_id < tx_hi)
+ if since is not None:
+ stmt = stmt.where(tx_tbl.c.issued_at >= since)
+ if until is not None:
+ stmt = stmt.where(tx_tbl.c.issued_at < until)
+ # Match fetch_change_records' final Python sort key order
+ # (issued_at, transaction_id, sequence, entity_id) so the
+ # per-kind budget keeps exactly the rows the final sort ranks
+ # highest. entity_kind is constant within a per-kind statement.
+ stmt = stmt.order_by(
+ tx_tbl.c.issued_at.desc(),
+ vc.c.transaction_id.desc(),
+ vc.c.sequence.desc(),
+ vc.c.entity_id.desc(),
+ ).limit(remaining)
+ kind_rows.extend(
+ dict(row)
+ for row in db.session.connection().execute(stmt).mappings().all()
+ )
Review Comment:
Addressed by subsequent commits; the flagged code is no longer current after the rebase onto master. Resolving as outdated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(versioning): cross-entity version activity view [superset]
Posted by "mikebridge (via GitHub)" <gi...@apache.org>.
mikebridge commented on code in PR #41076:
URL: https://github.com/apache/superset/pull/41076#discussion_r3574961163
##########
superset/migrations/versions/2026-05-01_23-36_2bee73611e32_composite_pk_association_tables.py:
##########
@@ -0,0 +1,580 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""composite_pk_association_tables
+
+Replace the unused synthetic ``id INTEGER PRIMARY KEY`` on eight many-to-many
+association tables with a composite primary key on the two FK columns. Drops
+the now-redundant ``UniqueConstraint(fk1, fk2)`` on the two tables that
+already carry one. Pre-flight: deletes rows with NULL FK values (six tables
+allow them today) and any duplicate ``(fk1, fk2)`` rows.
+
+Motivated by SQLAlchemy-Continuum issue #129 (M2M restore against junction
+tables with surrogate PKs); also closes the data-integrity hole where six
+of the eight tables lacked DB-level uniqueness.
+
+Revision ID: 2bee73611e32
+Revises: 78a40c08b4be
+Create Date: 2026-05-01 23:36:34.050058
+
+"""
+
+import logging
+from typing import NamedTuple
+
+import sqlalchemy as sa
+from alembic import op
+from alembic.operations.base import BatchOperations
+from sqlalchemy import inspect
+from sqlalchemy.engine import Connection
+
+# revision identifiers, used by Alembic.
+revision = "2bee73611e32"
+down_revision = "78a40c08b4be"
+
+logger = logging.getLogger("alembic.env")
+
+
+class AssociationTable(NamedTuple):
+ """A junction table being converted from surrogate-id PK to composite-FK PK."""
+
+ name: str
+ fk1: str
+ fk2: str
+
+
+# Order is alphabetical by table name; deterministic for review and bisection.
+AFFECTED_TABLES: list[AssociationTable] = [
+ AssociationTable("dashboard_roles", "dashboard_id", "role_id"),
+ AssociationTable("dashboard_slices", "dashboard_id", "slice_id"),
+ AssociationTable("dashboard_user", "user_id", "dashboard_id"),
+ AssociationTable("report_schedule_user", "user_id", "report_schedule_id"),
+ AssociationTable("rls_filter_roles", "role_id", "rls_filter_id"),
+ AssociationTable("rls_filter_tables", "table_id", "rls_filter_id"),
+ AssociationTable("slice_user", "user_id", "slice_id"),
+ AssociationTable("sqlatable_user", "user_id", "table_id"),
+]
+
+# These two tables already declare ``UniqueConstraint(fk1, fk2)`` in the model;
+# the composite PK subsumes it, so the migration drops the redundant constraint.
+TABLES_WITH_PRE_EXISTING_UNIQUE: set[str] = {
+ "dashboard_slices",
+ "report_schedule_user",
+}
+
+# Documentation set: tables whose FK columns are nullable in their original
+# create_table migrations (``dashboard_roles.dashboard_id`` from revision
+# e11ccdd12658 is the most recent addition). ``report_schedule_user`` is the
+# only affected table created with both FK columns ``NOT NULL`` and is
+# intentionally absent here. This set is no longer consulted at runtime — the
+# upgrade now runs the NULL-FK cleanup on every affected table because the
+# DELETE is a cheap no-op when the columns are already NOT NULL, and that
+# eliminates the risk of bugs from this set going stale (the
+# ``dashboard_roles`` omission caught in PR review was exactly that bug).
+TABLES_WITH_NULLABLE_FKS: set[str] = {
+ "dashboard_roles",
+ "dashboard_slices",
+ "dashboard_user",
+ "rls_filter_roles",
+ "rls_filter_tables",
+ "slice_user",
+ "sqlatable_user",
+}
+
+
+def _check_no_external_fks_to_id(conn: Connection) -> None:
+ """Raise ``RuntimeError`` if any foreign key in the database references one
+ of the eight junction-table ``id`` columns. Uses SQLAlchemy's ``Inspector``
+ for dialect-agnostic introspection across PostgreSQL, MySQL, and SQLite.
+
+ Scope limitation: ``Inspector.get_table_names()`` returns tables in the
+ connection's default schema only. On PostgreSQL deployments where Superset
+ metadata lives in a non-default schema, or on multi-schema deployments
+ that allow cross-schema FKs, an external FK in another schema would not
+ be detected. This is acceptable for the standard single-schema
+ deployment that Superset documents; operators with multi-schema
+ metadata should run the equivalent inventory query against
+ ``information_schema.referential_constraints`` themselves before
+ applying.
+ """
+ affected = {t.name for t in AFFECTED_TABLES}
+ insp = inspect(conn)
+ for table_name in insp.get_table_names():
+ if table_name in affected:
+ continue
+ for fk in insp.get_foreign_keys(table_name):
+ if fk["referred_table"] in affected and "id" in fk["referred_columns"]:
+ raise RuntimeError(
+ f"Cannot drop synthetic id from {fk['referred_table']}: "
+ f"external FK {fk.get('name', '<unnamed>')} on {table_name} "
+ f"references {fk['referred_table']}({fk['referred_columns']}). "
+ "Drop or migrate the referencing FK before applying this "
+ "migration."
+ )
+
+
+def _table_clause(t: AssociationTable) -> sa.sql.expression.TableClause:
+ """Build a lightweight SQLAlchemy ``TableClause`` for ``t`` exposing the
+ columns the helper queries reference (``id``, ``fk1``, ``fk2``). Used so
+ that the dedupe / cleanup / assert SQL can be expressed via SQLAlchemy
+ core constructs rather than via string interpolation."""
+ return sa.table(t.name, sa.column("id"), sa.column(t.fk1), sa.column(t.fk2))
+
+
+def _delete_null_fk_rows(conn: Connection, t: AssociationTable) -> int:
+ """Delete rows where ``t.fk1`` or ``t.fk2`` is NULL on ``t.name``.
+
+ Returns the deletion count. Required because primary-key columns must be
+ NOT NULL; the PK-add downstream would fail with a cryptic constraint
+ violation if any NULL-FK rows survived. Run unconditionally on every
+ affected table — see ``TABLES_WITH_NULLABLE_FKS`` above for the rationale.
+ """
+ tbl = _table_clause(t)
+ stmt = sa.delete(tbl).where(sa.or_(tbl.c[t.fk1].is_(None), tbl.c[t.fk2].is_(None)))
+ result = conn.execute(stmt)
+ n = result.rowcount or 0
+ if n:
+ logger.warning(
+ "Deleted %d row(s) with NULL FK from %s before composite-PK promotion",
+ n,
+ t.name,
+ )
+ return n
+
+
+def _dedupe_by_min_id(conn: Connection, t: AssociationTable) -> int:
+ """Delete duplicate ``(t.fk1, t.fk2)`` rows from ``t.name`` keeping ``MIN(id)``.
+
+ Returns the deletion count. The ``NOT IN`` argument is wrapped in an
+ extra ``SELECT keep_id FROM (...) AS s`` derived table because MySQL
+ rejects ``DELETE FROM t WHERE id NOT IN (SELECT MIN(id) FROM t GROUP BY
+ ...)`` with ERROR 1093 unless the inner SELECT is materialized through
+ a derived table. SQLAlchemy's ``.subquery()`` produces that wrap.
+
+ Logs a sample (up to 10) of the discarded ``(fk1, fk2, id)`` tuples at
+ WARN before deletion, so operators can audit which rows are dropped —
+ the "keep ``MIN(id)``" policy preserves the original row, which is
+ correct in practice but discards any later, semantically-identical
+ re-grants.
+ """
+ tbl = _table_clause(t)
+
+ keep_min = (
+ sa.select(sa.func.min(tbl.c.id).label("keep_id"))
+ .group_by(tbl.c[t.fk1], tbl.c[t.fk2])
+ .subquery("keep_min")
+ )
+ keep_ids = sa.select(keep_min.c.keep_id)
+ discarded = tbl.c.id.notin_(keep_ids)
+
+ sample_stmt = (
+ sa.select(tbl.c[t.fk1], tbl.c[t.fk2], tbl.c.id).where(discarded).limit(10)
+ )
+ sample = list(conn.execute(sample_stmt))
+
+ delete_stmt = sa.delete(tbl).where(discarded)
+ result = conn.execute(delete_stmt)
+ n = result.rowcount or 0
+ if n:
+ logger.warning(
+ "Deduped %d duplicate row(s) from %s; sample of discarded "
+ "(%s, %s, id) tuples (up to 10): %s",
+ n,
+ t.name,
+ t.fk1,
+ t.fk2,
+ sample,
+ )
+ return n
+
+
+def _assert_no_duplicates(conn: Connection, t: AssociationTable) -> None:
+ """Raise ``RuntimeError`` if any ``(t.fk1, t.fk2)`` duplicate group remains.
+
+ Called after ``_dedupe_by_min_id`` to surface silent dialect-dependent
+ dedupe failures (e.g., a MySQL syntax issue) as an actionable error
+ before the PK-add fires with a less-helpful constraint-violation message.
+ """
+ tbl = _table_clause(t)
+ duplicate_groups = (
+ sa.select(sa.literal(1))
+ .select_from(tbl)
+ .group_by(tbl.c[t.fk1], tbl.c[t.fk2])
+ .having(sa.func.count() > 1)
+ .subquery("duplicate_groups")
+ )
+ count_stmt = sa.select(sa.func.count()).select_from(duplicate_groups)
+ if remaining := conn.scalar(count_stmt) or 0:
+ raise RuntimeError(
+ f"Dedupe failed for {t.name}: {remaining} duplicate "
+ f"({t.fk1}, {t.fk2}) groups remain after _dedupe_by_min_id. "
+ f"Check the dedupe SQL for dialect {conn.dialect.name}."
+ )
+
+
+def _build_pre_upgrade_table(
+ insp: sa.engine.reflection.Inspector,
+ t: AssociationTable,
+ fks: list[dict] | None = None,
+) -> sa.Table:
+ """Build a ``Table`` object representing the pre-upgrade schema of ``t``,
+ explicitly *without* any redundant ``UniqueConstraint(t.fk1, t.fk2)``.
+ Used as ``copy_from`` to ``batch_alter_table`` so the rebuilt table
+ omits the unnamed UNIQUE constraint deterministically across dialects
+ (SQLite reflects unnamed UNIQUEs with ``name=None``, defeating the
+ standard ``batch_op.drop_constraint(name)`` path).
+
+ Reflects column types and FK targets (with original FK constraint names
+ preserved) from the live database; only the redundant UNIQUE is omitted.
+
+ *fks* lets a caller pass a pre-captured ``get_foreign_keys`` result.
+ The MySQL upgrade path drops the live FK constraints before building
+ this table, so re-reflecting here would only see them via the
+ Inspector's per-instance ``info_cache`` — an implementation detail,
+ not a contract. Passing the pre-drop list makes the dependency
+ explicit instead of relying on reflection caching.
+ """
+ md = sa.MetaData()
+ if fks is None:
+ fks = insp.get_foreign_keys(t.name)
+ fks_for_col: dict[str, list[dict]] = {}
+ for fk in fks:
+ for col_name in fk["constrained_columns"]:
+ fks_for_col.setdefault(col_name, []).append(fk)
+
+ cols: list[sa.Column] = []
+ for c in insp.get_columns(t.name):
+ col_kwargs = {"nullable": c.get("nullable", True)}
+ if c["name"] == "id":
+ col_kwargs["primary_key"] = True
+ col_kwargs["autoincrement"] = True
+ fk_args = []
+ for fk in fks_for_col.get(c["name"], []):
+ idx = fk["constrained_columns"].index(c["name"])
+ target = f"{fk['referred_table']}.{fk['referred_columns'][idx]}"
+ options = {}
+ if fk.get("options", {}).get("ondelete"):
+ options["ondelete"] = fk["options"]["ondelete"]
+ if fk.get("name"):
+ options["name"] = fk["name"]
+ fk_args.append(sa.ForeignKey(target, **options))
+ cols.append(sa.Column(c["name"], c["type"], *fk_args, **col_kwargs))
+ return sa.Table(t.name, md, *cols)
+
+
+def _drop_redundant_unique_by_name(
+ conn: Connection, insp: sa.engine.reflection.Inspector, t: AssociationTable
+) -> None:
+ """Drop the redundant ``UNIQUE(fk1, fk2)`` constraint by its reflected
+ name on PostgreSQL / MySQL.
+
+ The two tables in ``TABLES_WITH_PRE_EXISTING_UNIQUE`` carry a UNIQUE
+ constraint that the composite primary key subsumes. PostgreSQL and
+ MySQL both auto-name UNIQUE constraints (``<table>_<cols>_key`` on
+ Postgres, ``<table>_<col>_<n>`` or the explicit ``uq_*`` we may have
+ given it on MySQL), so they're reflectable by name. SQLite is
+ handled separately via ``recreate="always"`` + ``copy_from`` because
+ it reflects unnamed UNIQUEs with ``name=None``.
+
+ No-op if no matching UNIQUE is found (defensive — re-runs after a
+ partial application should not error).
+ """
+ for uc in insp.get_unique_constraints(t.name):
+ if set(uc.get("column_names", [])) == {t.fk1, t.fk2} and uc.get("name"):
+ op.drop_constraint(uc["name"], t.name, type_="unique")
+ return
+
+
+# MySQL ON DELETE actions that the downgrade re-create loop is allowed
+# to interpolate into raw SQL. The reflected value comes from MySQL's
+# information_schema (so not user input), but a whitelist eliminates
+# the "what if an unexpected value appears" question entirely. The
+# four entries are the SQL-standard set; SET DEFAULT is intentionally
+# excluded because InnoDB silently downgrades it to NO ACTION.
+_VALID_ONDELETE_ACTIONS: frozenset[str] = frozenset(
+ {"CASCADE", "SET NULL", "RESTRICT", "NO ACTION"}
+)
+
+
+def _enforce_not_null_for_sqlite(
+ batch_op: BatchOperations, t: AssociationTable, conn: Connection
+) -> None:
+ """Force ``NOT NULL`` on the FK columns post-PK-promotion on SQLite only.
+
+ SQLite has a long-standing quirk: composite ``PRIMARY KEY`` does not
+ promote constituent columns to ``NOT NULL`` (only ``INTEGER PRIMARY KEY``
+ does). PostgreSQL and MySQL implicitly promote the PK columns to
+ ``NOT NULL`` when the constraint is added, making the explicit
+ ``alter_column`` redundant there.
+
+ Skipping the ``alter_column`` on MySQL is also functionally required:
+ MySQL 8 rejects ``ALTER COLUMN`` on a column that participates in a
+ foreign key constraint with ``ERROR 1832 (HY000): Cannot change column
+ 'X': used in a foreign key constraint 'Y'`` whenever the table has
+ data — even when the only change is ``NULL`` → ``NOT NULL`` and the
+ column is already part of a freshly-added composite primary key (which
+ InnoDB has just made implicitly ``NOT NULL`` anyway). The error fires
+ on populated tables but not on empty ones, which is why CI's
+ ``test-mysql`` shard (fresh schema) didn't catch this and a real
+ production-shaped install does.
+
+ Only SQLite still needs the explicit step, and SQLite has no FK
+ enforcement objection.
+ """
+ if conn.dialect.name == "sqlite":
+ batch_op.alter_column(t.fk1, existing_type=sa.Integer, nullable=False)
+ batch_op.alter_column(t.fk2, existing_type=sa.Integer, nullable=False)
+
+
+def upgrade() -> None:
+ conn = op.get_bind()
+ _check_no_external_fks_to_id(conn)
+ insp = inspect(conn)
+
+ for t in AFFECTED_TABLES:
+ # Resumability guard: on MySQL every DDL statement auto-commits, so
+ # a failure at table N of 8 leaves tables 1..N-1 already converted
+ # while ``alembic_version`` is still un-stamped. Without this guard
+ # a re-run would fail at table 1 (``drop_column("id")`` on a table
+ # that no longer has ``id``), and ``downgrade`` can't run either
+ # (the revision was never stamped) — recovery would need manual
+ # surgery. A converted table is identified by the absent ``id``
+ # column; skipping it makes re-running the upgrade safe on every
+ # dialect (Postgres/SQLite wrap the migration in a transaction, so
+ # the guard is simply never hit there).
+ if "id" not in {c["name"] for c in insp.get_columns(t.name)}:
+ logger.info(
+ "%s: already converted (no surrogate id column); skipping",
+ t.name,
+ )
+ continue
+
+ # Run NULL-FK cleanup unconditionally: it is a no-op DELETE on tables
+ # whose FK columns are already NOT NULL (cheap), and skipping it on a
+ # table whose FK was nullable would leave the PK-add to fail with a
+ # cryptic constraint violation. Cf. ``TABLES_WITH_NULLABLE_FKS`` above
+ # for documentation of which tables are known to have nullable FKs.
+ _delete_null_fk_rows(conn, t)
+ _dedupe_by_min_id(conn, t)
+ _assert_no_duplicates(conn, t)
+
+ # Two tables (``dashboard_slices``, ``report_schedule_user``)
+ # carry a redundant ``UNIQUE(fk1, fk2)`` that the composite PK
+ # subsumes. Three dialect-specific paths:
+ #
+ # * **PostgreSQL** — the UNIQUE constraint has a stable
+ # reflected name (Postgres default convention), so we
+ # ``DROP CONSTRAINT`` by name and then run the structural
+ # change as direct ALTER. This avoids the full-table copy
+ # that ``recreate="always"`` would trigger
+ # (``CREATE TABLE AS SELECT → DROP → RENAME``), holding
+ # ``ACCESS EXCLUSIVE`` only for the (much shorter) PK
+ # index build instead of the full copy duration.
+ #
+ # * **MySQL** — InnoDB binds the FK constraints to the
+ # redundant UNIQUE's underlying index for back-reference,
+ # so a direct ``DROP CONSTRAINT`` of the UNIQUE raises
+ # ``ERROR 1553``. Use ``recreate="always"`` to rebuild the
+ # table without the UNIQUE; drop the FKs first to dodge
+ # the ``ERROR 1826`` (duplicate FK constraint name) that
+ # the temp-table phase would otherwise provoke. The FKs
+ # are re-created automatically as part of ``copy_from``.
+ #
+ # * **SQLite** — unnamed UNIQUE constraints reflect with
+ # ``name=None`` and can't be dropped by name. Use
+ # ``recreate="always"`` + ``copy_from`` (omits UNIQUE).
+ # SQLite always rebuilds for PK changes anyway, so the
+ # recreate isn't extra cost there.
+ if t.name in TABLES_WITH_PRE_EXISTING_UNIQUE:
+ if conn.dialect.name == "postgresql":
+ _drop_redundant_unique_by_name(conn, insp, t)
+ with op.batch_alter_table(t.name) as batch_op:
+ batch_op.drop_column("id")
+ batch_op.create_primary_key(f"pk_{t.name}", [t.fk1, t.fk2])
+ _enforce_not_null_for_sqlite(batch_op, t, conn)
+ else:
+ # Capture the FK list BEFORE dropping: the copy_from table
+ # below must embed these constraints, and re-reflecting
+ # after the drop only works via the Inspector's
+ # per-instance info_cache (see _build_pre_upgrade_table).
+ pre_drop_fks = insp.get_foreign_keys(t.name)
+ if conn.dialect.name == "mysql":
+ for fk in pre_drop_fks:
+ if fk_name := fk.get("name"):
+ op.drop_constraint(fk_name, t.name, type_="foreignkey")
+ with op.batch_alter_table(
+ t.name,
+ recreate="always",
+ copy_from=_build_pre_upgrade_table(insp, t, fks=pre_drop_fks),
+ ) as batch_op:
+ batch_op.drop_column("id")
+ batch_op.create_primary_key(f"pk_{t.name}", [t.fk1, t.fk2])
+ _enforce_not_null_for_sqlite(batch_op, t, conn)
+ else:
+ with op.batch_alter_table(t.name) as batch_op:
+ batch_op.drop_column("id")
+ batch_op.create_primary_key(f"pk_{t.name}", [t.fk1, t.fk2])
+ _enforce_not_null_for_sqlite(batch_op, t, conn)
+
+
+def downgrade() -> None:
+ # Inverse order: undo upgrade transformations from last-applied to
+ # first-applied. Within each table, drop the composite PK, restore the
+ # surrogate ``id`` column, and re-add the original ``UNIQUE`` constraint
+ # on the two tables that previously carried one.
+ #
+ # Note: FK columns remain NOT NULL after downgrade (intentional asymmetry
+ # — see UPDATING.md). Restoring the original nullable state would require
+ # an explicit ``alter_column`` per FK per table for no operator value;
+ # junction-table NULL FKs were always meaningless under ``secondary=``
+ # semantics.
+ # The downgrade names the restored PK ``<table>_pkey`` (matching Postgres'
+ # default constraint-naming convention, which was the original constraint
+ # name before this migration ran) so a downgrade-then-upgrade round-trip
+ # doesn't collide on the upgrade's ``pk_<table>`` name.
+ #
+ # Adding a NOT NULL ``id`` column to a table with existing rows requires
+ # a default that fires on the existing rows. ``sa.Identity()`` (Postgres
+ # 10+ / MySQL 8+) and ``sa.Sequence`` (with explicit nextval) both
+ # backfill existing rows during ALTER TABLE; bare ``autoincrement=True``
+ # does not. ``Identity`` is the modern portable choice.
+ conn = op.get_bind()
+ insp = inspect(conn)
+ is_mysql = conn.dialect.name == "mysql"
+ for t in reversed(AFFECTED_TABLES):
+ if is_mysql:
+ _downgrade_mysql_table(insp, t)
+ else:
+ with op.batch_alter_table(t.name) as batch_op:
+ batch_op.drop_constraint(f"pk_{t.name}", type_="primary")
+ batch_op.add_column(
+ sa.Column(
+ "id",
+ sa.Integer,
+ sa.Identity(always=False),
+ nullable=False,
+ )
+ )
+ batch_op.create_primary_key(f"{t.name}_pkey", ["id"])
+ if t.name in TABLES_WITH_PRE_EXISTING_UNIQUE:
+ batch_op.create_unique_constraint(
+ f"uq_{t.name}_{t.fk1}_{t.fk2}", [t.fk1, t.fk2]
+ )
+
+
+def _downgrade_mysql_table(
+ insp: sa.engine.reflection.Inspector, t: AssociationTable
+) -> None:
+ """MySQL-specific downgrade for one table.
+
+ Two MySQL quirks force a dialect-specific path here:
+
+ 1. **ERROR 1553 — ``Cannot drop index 'PRIMARY': needed in a foreign
+ key constraint``**. InnoDB uses the composite PK index to back the
+ FK on the leftmost column. Dropping the PK before the FKs orphans
+ that backing index. PostgreSQL and SQLite create separate indexes
+ for FK columns and don't need this dance. We drop the FKs first
+ and re-add them after the structural change.
+
+ 2. **``Identity(always=False)`` on a non-PK column add does not emit
+ ``AUTO_INCREMENT`` on MySQL.** SQLAlchemy 1.4 only emits
+ ``AUTO_INCREMENT`` when the column has both ``Identity()`` and
+ ``primary_key=True`` at create time. Our portable path adds the
+ column first, then creates the PK separately — which works on
+ Postgres (the column gets ``GENERATED BY DEFAULT AS IDENTITY``)
+ and SQLite (``INTEGER PRIMARY KEY`` becomes a rowid alias) but
+ leaves MySQL without auto-generation, so existing rows can't be
+ backfilled and future ``INSERT`` statements fail with
+ ``Field 'id' doesn't have a default value``. The combined
+ ``DROP PRIMARY KEY, ADD COLUMN AUTO_INCREMENT, ADD PRIMARY KEY``
+ in a single ALTER statement is the canonical MySQL idiom: MySQL
+ backfills existing rows with sequential values and the column
+ remains auto-incrementing for future inserts.
+
+ Raw SQL is unavoidable here — there is no SQLAlchemy core equivalent
+ for the combined-ALTER form, and the constitution allows raw SQL for
+ dialect-specific DDL with no programmatic equivalent (preferring
+ triple-quoted strings for legibility).
+
+ Belt-and-braces guard: ``t.name`` is interpolated as a backtick-quoted
+ identifier in the ALTER statements below. The value comes from
+ ``AFFECTED_TABLES`` (a module-level literal), so SQL injection is
+ structurally precluded. The explicit ``allowed`` check here makes
+ that invariant load-bearing rather than implicit, so a future
+ refactor that loosens the call-site can't slip past review.
+ """
+ allowed = {a.name for a in AFFECTED_TABLES}
+ if t.name not in allowed:
+ raise RuntimeError(
+ f"Refusing to ALTER unknown table {t.name!r}: "
+ f"only AFFECTED_TABLES entries may flow through this path."
+ )
+
+ fks = insp.get_foreign_keys(t.name)
+
+ for fk in fks:
+ if fk_name := fk.get("name"):
+ op.execute(f"ALTER TABLE `{t.name}` DROP FOREIGN KEY `{fk_name}`")
Review Comment:
Addressed by subsequent commits; the flagged code is no longer current after the rebase onto master. Resolving as outdated.
##########
superset/migrations/versions/2026-05-28_19-50_56cd24c07170_add_versioning_tables.py:
##########
@@ -0,0 +1,562 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""add_versioning_tables
+
+Creates the full schema backing entity versioning in a single
+migration:
+
+1. ``version_transaction`` — audit log keyed by Continuum's per-flush
+ transaction id (plus a Postgres-specific id sequence).
+2. **Parent shadow tables** mirroring each versioned entity's columns:
+ ``dashboards_version`` / ``slices_version`` / ``tables_version``.
+3. ``version_changes`` — field-level diff log keyed to a
+ ``(transaction, entity)`` pair; each row describes one atomic change
+ (one field or one child-collection element) that occurred during a
+ save.
+4. **Child shadow tables** for the collections Continuum auto-registers
+ when ``__versioned__`` is applied to ``TableColumn`` / ``SqlMetric``
+ and the ``slices`` exclude is removed from
+ ``Dashboard.__versioned__``: ``table_columns_version`` /
+ ``sql_metrics_version`` / ``dashboard_slices_version``.
+
+All shadow tables follow the validity-strategy shape (mirrored columns
++ ``transaction_id`` / ``end_transaction_id`` / ``operation_type``
+bookkeeping with FKs to ``version_transaction.id``). The current
+version row has ``end_transaction_id = NULL``.
+
+This migration replaces three iterative migrations from the spike phase
+(``56cd24c07170``, ``e1f3c5a7b9d0``, ``f7a2b3c4d5e6``) that captured the
+same schema in three steps as the feature was developed. Compacting
+gives downstream operators one migration to apply / reverse and one
+review surface. The ``revision`` hash is reused from the original first
+migration so anyone still tracking the chain by that hash lands on the
+same logical change set.
+
+Generated by hand because the current Continuum + Alembic-autogenerate
+interaction trips on the renamed ``transaction`` -> ``version_transaction``
+table key (``KeyError`` lookups in ``table_key_to_table``). Column
+inventories were sourced from the live model ``__table__`` definitions
+and ``version_class(...).__table__`` / Continuum association metadata.
+
+Primary key choice. Both ``version_transaction.id`` and
+``version_changes.id`` are ``BigInteger`` autoincrement — a deliberate
+carveout from the project's UUID-PK convention for new models (see
+``CLAUDE.md`` §"UUID Migration"). ``version_transaction`` is keyed
+externally by SQLAlchemy-Continuum via
+``nextval('version_transaction_id_seq')`` on every INSERT; matching
+that contract is required for ``versioning_manager`` to function.
+``version_changes`` follows the same shape because the user-facing
+identity is the ``(transaction_id, entity_kind, entity_id, sequence)``
+composite unique key, not the row id; the API surfaces a deterministic
+UUIDv5 ``version_uuid`` derived from ``entity.uuid`` and
+``transaction_id`` for stable external references.
+
+Revision ID: 56cd24c07170
+Revises: 2bee73611e32
+Create Date: 2026-05-28 19:50:00.000000
+
+"""
+
+from __future__ import annotations
+
+import sqlalchemy as sa
+from alembic import op
+from sqlalchemy_utils import UUIDType
+
+from superset.utils.core import MediumText
+
+revision = "56cd24c07170"
+# Stacked on the composite-PK association-tables change (2bee73611e32) so the
+# Continuum shadow tables this migration creates can mirror the
+# composite-PK shape of the live association tables. If that change
+# is removed from the stack, this should be reverted to "ce6bd21901ab".
+down_revision = "2bee73611e32"
+
+
+def upgrade() -> None:
+ bind = op.get_bind()
+
+ # ------------------------------------------------------------------
+ # version_transaction
+ #
+ # Audit log for each versioning event. Continuum emits
+ # ``nextval('version_transaction_id_seq')`` on every INSERT, so the
+ # sequence must exist before the table on Postgres. SQLite/MySQL
+ # ignore the explicit CREATE SEQUENCE (they auto-increment natively).
+ # ------------------------------------------------------------------
+ if bind.dialect.name == "postgresql":
+ op.execute("CREATE SEQUENCE IF NOT EXISTS version_transaction_id_seq")
+
+ op.create_table(
Review Comment:
Addressed by subsequent commits; the flagged code is no longer current after the rebase onto master. Resolving as outdated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(versioning): cross-entity version activity view [superset]
Posted by "codeant-ai-for-open-source[bot] (via GitHub)" <gi...@apache.org>.
codeant-ai-for-open-source[bot] commented on code in PR #41076:
URL: https://github.com/apache/superset/pull/41076#discussion_r3500128655
##########
superset/commands/dashboard/importers/v1/__init__.py:
##########
@@ -201,7 +205,31 @@ def _import(
overwrite
or (dashboard.id, chart_id) not in existing_relationships
):
- dashboard_chart_ids.append((dashboard.id, chart_id))
+ target_chart_ids.append(chart_id)
+
+ if overwrite:
+ # Replace the dashboard's chart membership entirely.
+ dashboard.slices = (
+ db.session.query(Slice)
+ .filter(Slice.id.in_(target_chart_ids))
+ .all()
+ if target_chart_ids
+ else []
+ )
+ # Flush eagerly so the M2M rows land in
+ # ``dashboard_slices`` before any subsequent
+ # autoflush fires an inner-flush event handler
+ # that would reset the relationship change.
+ db.session.flush()
+ elif target_chart_ids:
+ # Append only the new associations to existing ones.
+ new_slices = (
+ db.session.query(Slice)
+ .filter(Slice.id.in_(target_chart_ids))
+ .all()
+ )
Review Comment:
**Suggestion:** Add a type annotation for the newly introduced local collection so it complies with the required type-hint coverage for annotatable variables. [custom_rule]
**Severity Level:** Minor ⚠️
<details>
<summary><b>Why it matters? 🤔 </b></summary>
The new local variable `new_slices` is introduced without a type annotation even though its type is readily inferable (`list[Slice]`). This matches the rule requiring type hints for annotatable new or modified Python variables.
</details>
[](https://app.codeant.ai/fix-in-ide?tool=cursor&prompt_id=2a91d65e6300498ca9874eefc4b016c6&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset) [](https://app.codeant.ai/fix-in-ide?tool=vscode-claude&prompt_id=2a91d65e6300498ca9874eefc4b016c6&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset)
*(Use Cmd/Ctrl + Click for best experience)*
<details>
<summary><b>Prompt for AI Agent 🤖 </b></summary>
```mdx
This is a comment left during a code review.
**Path:** superset/commands/dashboard/importers/v1/__init__.py
**Line:** 226:230
**Comment:**
*Custom Rule: Add a type annotation for the newly introduced local collection so it complies with the required type-hint coverage for annotatable variables.
Validate the correctness of the flagged issue. If correct, How can I resolve this? If you propose a fix, implement it and please make it concise.
Once fix is implemented, also check other comments on the same PR, and ask user if the user wants to fix the rest of the comments as well. if said yes, then fetch all the comments validate the correctness and implement a minimal fix
```
</details>
<a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F41076&comment_hash=f46ef37da7497a70bace78bd83fb182e2ad4ffd60fdfd64748dbf796e82ca8b3&reaction=like'>👍</a> | <a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F41076&comment_hash=f46ef37da7497a70bace78bd83fb182e2ad4ffd60fdfd64748dbf796e82ca8b3&reaction=dislike'>👎</a>
##########
superset/commands/importers/v1/utils.py:
##########
@@ -387,6 +388,37 @@ def safe_insert_dashboard_chart_relationships(
)
+def _prime_versioning_unit_of_work() -> None:
+ """Ensure Continuum has a unit-of-work for the current connection.
+
+ ``dashboard_slices`` is a Continuum-tracked (versioned) association
+ table, so a raw Core INSERT/DELETE on it fires Continuum's engine-level
+ ``before_execute`` listener, which looks up a unit-of-work for the
+ connection and raises ``KeyError`` when none is registered (the same
+ failure class the dashboard test factory hit). The normal import flow
+ registers one via prior ORM flushes, so this is belt-and-suspenders for
+ a bulk relationship insert that might run before any flush on the
+ connection. No-op (the listener is detached) when version capture is
+ disabled, which is the shipped default; never allowed to break an import.
+ """
+ try:
+ # pylint: disable=import-outside-toplevel
+ from sqlalchemy_continuum import versioning_manager
+
+ # Mirror the exact condition Continuum's track_association_operations
+ # listener uses to decide whether it acts (versioning OR
+ # native_versioning), so the prime can't skip while the listener runs.
+ options = versioning_manager.options
Review Comment:
**Suggestion:** Add an explicit type annotation for this newly introduced local variable to satisfy the type-hint requirement for annotatable variables. [custom_rule]
**Severity Level:** Minor ⚠️
<details>
<summary><b>Why it matters? 🤔 </b></summary>
This is a newly added local variable in Python code, and it can be annotated with a type hint. Since the rule requires type hints on annotatable variables, the omission is a real violation.
</details>
[](https://app.codeant.ai/fix-in-ide?tool=cursor&prompt_id=d20de76a02b24619b73947504c07f268&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset) [](https://app.codeant.ai/fix-in-ide?tool=vscode-claude&prompt_id=d20de76a02b24619b73947504c07f268&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset)
*(Use Cmd/Ctrl + Click for best experience)*
<details>
<summary><b>Prompt for AI Agent 🤖 </b></summary>
```mdx
This is a comment left during a code review.
**Path:** superset/commands/importers/v1/utils.py
**Line:** 411:411
**Comment:**
*Custom Rule: Add an explicit type annotation for this newly introduced local variable to satisfy the type-hint requirement for annotatable variables.
Validate the correctness of the flagged issue. If correct, How can I resolve this? If you propose a fix, implement it and please make it concise.
Once fix is implemented, also check other comments on the same PR, and ask user if the user wants to fix the rest of the comments as well. if said yes, then fetch all the comments validate the correctness and implement a minimal fix
```
</details>
<a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F41076&comment_hash=7efe37ff4e64bc8dce1403903489a93fc0111df5325b783b565807f60be795e7&reaction=like'>👍</a> | <a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F41076&comment_hash=7efe37ff4e64bc8dce1403903489a93fc0111df5325b783b565807f60be795e7&reaction=dislike'>👎</a>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(versioning): cross-entity version activity view [superset]
Posted by "bito-code-review[bot] (via GitHub)" <gi...@apache.org>.
bito-code-review[bot] commented on PR #41076:
URL: https://github.com/apache/superset/pull/41076#issuecomment-4907568288
AI Code Review is in progress (usually takes 3 to 15 minutes unless it's a very large PR).
<details>
<summary>
Bito Usage Guide
</summary>
**Commands**
Type the following command in the pull request comment and save the comment.
- `/review` - Manually triggers a full AI review.
- `/pause` - Pauses automatic reviews on this pull request.
- `/resume` - Resumes automatic reviews.
- `/resolve` - Marks all Bito-posted review comments as resolved.
- `/abort` - Cancels all in-progress reviews.
Refer to the <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/available-commands" target="_blank" rel="noopener noreferrer">documentation</a> for additional commands.
**Configuration**
This repository uses `Superset` You can customize the agent settings <a href="https://alpha.bito.ai/home/ai-agents/code-review-agent" target="_blank" rel="noopener noreferrer">here</a> or contact your Bito workspace admin at evan@preset.io.
**Documentation & Help**
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/getting-started/install-run-using-bito-cloud/create-or-customize-an-agent-instance" target="_blank" rel="noopener noreferrer">Customize agent settings</a>
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/implementing-custom-code-review-rules" target="_blank" rel="noopener noreferrer">Review rules</a>
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent" target="_blank" rel="noopener noreferrer">General documentation</a>
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/faqs" target="_blank" rel="noopener noreferrer">FAQ</a>
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(versioning): cross-entity version activity view [superset]
Posted by "codecov[bot] (via GitHub)" <gi...@apache.org>.
codecov[bot] commented on PR #41076:
URL: https://github.com/apache/superset/pull/41076#issuecomment-4713557697
## [Codecov](https://app.codecov.io/gh/apache/superset/pull/41076?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:x: Patch coverage is `33.75451%` with `1835 lines` in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 55.41%. Comparing base ([`6e2db42`](https://app.codecov.io/gh/apache/superset/commit/6e2db42d9800eeb4c961860538b7c916d82c5a10?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`84ef36f`](https://app.codecov.io/gh/apache/superset/commit/84ef36f71f3919ecce2fe6529227a38d396fa8ef?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
| [Files with missing lines](https://app.codecov.io/gh/apache/superset/pull/41076?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [superset/versioning/activity/queries.py](https://app.codecov.io/gh/apache/superset/pull/41076?src=pr&el=tree&filepath=superset%2Fversioning%2Factivity%2Fqueries.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-c3VwZXJzZXQvdmVyc2lvbmluZy9hY3Rpdml0eS9xdWVyaWVzLnB5) | 0.00% | [176 Missing :warning: ](https://app.codecov.io/gh/apache/superset/pull/41076?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [superset/versioning/activity/orchestrator.py](https://app.codecov.io/gh/apache/superset/pull/41076?src=pr&el=tree&filepath=superset%2Fversioning%2Factivity%2Forchestrator.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-c3VwZXJzZXQvdmVyc2lvbmluZy9hY3Rpdml0eS9vcmNoZXN0cmF0b3IucHk=) | 0.00% | [137 Missing :warning: ](https://app.codecov.io/gh/apache/superset/pull/41076?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...end/src/pages/ChartList/VersionHistoryDropdown.tsx](https://app.codecov.io/gh/apache/superset/pull/41076?src=pr&el=tree&filepath=superset-frontend%2Fsrc%2Fpages%2FChartList%2FVersionHistoryDropdown.tsx&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-c3VwZXJzZXQtZnJvbnRlbmQvc3JjL3BhZ2VzL0NoYXJ0TGlzdC9WZXJzaW9uSGlzdG9yeURyb3Bkb3duLnRzeA==) | 2.27% | [129 Missing :warning: ](https://app.codecov.io/gh/apache/superset/pull/41076?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...src/pages/DashboardList/VersionHistoryDropdown.tsx](https://app.codecov.io/gh/apache/superset/pull/41076?src=pr&el=tree&filepath=superset-frontend%2Fsrc%2Fpages%2FDashboardList%2FVersionHistoryDropdown.tsx&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-c3VwZXJzZXQtZnJvbnRlbmQvc3JjL3BhZ2VzL0Rhc2hib2FyZExpc3QvVmVyc2lvbkhpc3RvcnlEcm9wZG93bi50c3g=) | 2.27% | [129 Missing :warning: ](https://app.codecov.io/gh/apache/superset/pull/41076?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...d/src/pages/DatasetList/VersionHistoryDropdown.tsx](https://app.codecov.io/gh/apache/superset/pull/41076?src=pr&el=tree&filepath=superset-frontend%2Fsrc%2Fpages%2FDatasetList%2FVersionHistoryDropdown.tsx&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-c3VwZXJzZXQtZnJvbnRlbmQvc3JjL3BhZ2VzL0RhdGFzZXRMaXN0L1ZlcnNpb25IaXN0b3J5RHJvcGRvd24udHN4) | 2.27% | [129 Missing :warning: ](https://app.codecov.io/gh/apache/superset/pull/41076?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [superset/versioning/diff.py](https://app.codecov.io/gh/apache/superset/pull/41076?src=pr&el=tree&filepath=superset%2Fversioning%2Fdiff.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-c3VwZXJzZXQvdmVyc2lvbmluZy9kaWZmLnB5) | 60.82% | [87 Missing and 18 partials :warning: ](https://app.codecov.io/gh/apache/superset/pull/41076?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [superset/versioning/queries.py](https://app.codecov.io/gh/apache/superset/pull/41076?src=pr&el=tree&filepath=superset%2Fversioning%2Fqueries.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-c3VwZXJzZXQvdmVyc2lvbmluZy9xdWVyaWVzLnB5) | 22.13% | [95 Missing :warning: ](https://app.codecov.io/gh/apache/superset/pull/41076?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...uperset-frontend/src/pages/ActivityDebug/index.tsx](https://app.codecov.io/gh/apache/superset/pull/41076?src=pr&el=tree&filepath=superset-frontend%2Fsrc%2Fpages%2FActivityDebug%2Findex.tsx&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-c3VwZXJzZXQtZnJvbnRlbmQvc3JjL3BhZ2VzL0FjdGl2aXR5RGVidWcvaW5kZXgudHN4) | 0.00% | [93 Missing :warning: ](https://app.codecov.io/gh/apache/superset/pull/41076?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [superset/versioning/activity/render.py](https://app.codecov.io/gh/apache/superset/pull/41076?src=pr&el=tree&filepath=superset%2Fversioning%2Factivity%2Frender.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-c3VwZXJzZXQvdmVyc2lvbmluZy9hY3Rpdml0eS9yZW5kZXIucHk=) | 0.00% | [76 Missing :warning: ](https://app.codecov.io/gh/apache/superset/pull/41076?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [superset/versioning/api\_helpers.py](https://app.codecov.io/gh/apache/superset/pull/41076?src=pr&el=tree&filepath=superset%2Fversioning%2Fapi_helpers.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-c3VwZXJzZXQvdmVyc2lvbmluZy9hcGlfaGVscGVycy5weQ==) | 32.55% | [58 Missing :warning: ](https://app.codecov.io/gh/apache/superset/pull/41076?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| ... and [42 more](https://app.codecov.io/gh/apache/superset/pull/41076?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | |
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #41076 +/- ##
==========================================
- Coverage 64.33% 55.41% -8.93%
==========================================
Files 2651 2695 +44
Lines 144652 147995 +3343
Branches 33371 33951 +580
==========================================
- Hits 93066 82010 -11056
- Misses 49935 65196 +15261
+ Partials 1651 789 -862
```
| [Flag](https://app.codecov.io/gh/apache/superset/pull/41076/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [hive](https://app.codecov.io/gh/apache/superset/pull/41076/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `38.94% <26.19%> (-0.41%)` | :arrow_down: |
| [javascript](https://app.codecov.io/gh/apache/superset/pull/41076/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `68.04% <2.98%> (-0.43%)` | :arrow_down: |
| [mysql](https://app.codecov.io/gh/apache/superset/pull/41076/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [postgres](https://app.codecov.io/gh/apache/superset/pull/41076/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [presto](https://app.codecov.io/gh/apache/superset/pull/41076/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `40.97% <40.56%> (+0.04%)` | :arrow_up: |
| [python](https://app.codecov.io/gh/apache/superset/pull/41076/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `41.03% <40.56%> (-18.57%)` | :arrow_down: |
| [sqlite](https://app.codecov.io/gh/apache/superset/pull/41076/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [superset-extensions-cli](https://app.codecov.io/gh/apache/superset/pull/41076/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `90.59% <ø> (?)` | |
| [unit](https://app.codecov.io/gh/apache/superset/pull/41076/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
</details>
[:umbrella: View full report in Codecov by Harness](https://app.codecov.io/gh/apache/superset/pull/41076?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
- :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(versioning): cross-entity version activity view [superset]
Posted by "mikebridge (via GitHub)" <gi...@apache.org>.
mikebridge commented on code in PR #41076:
URL: https://github.com/apache/superset/pull/41076#discussion_r3575001293
##########
superset/versioning/activity/render.py:
##########
@@ -0,0 +1,268 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Decoration: turn raw change-records into the ActivityRecord DTO.
+
+After fetching + filtering, each record needs the synthesized fields
+the API contract documents — ``entity_kind`` translated to the user-
+facing form, ``entity_uuid``, ``entity_deleted`` /
+``entity_deletion_state``, ``source`` (self vs. related),
+``summary`` (the headline), ``impact`` (chart-count for
+dashboard→dataset records), ``version_uuid``, ``changed_by``.
+
+This module collects all those decorations:
+
+* :func:`apply_record_decoration` — orchestrates the per-page additions in
+ one pass: pulls tombstones + uuids + impact counts in batches, then
+ walks records adding the synthesized fields and stripping the
+ internal-only columns the API contract doesn't expose.
+* :func:`_lookup_entity_uuids` — one IN-clause query per kind to
+ resolve live ``uuid`` for non-tombstoned entities.
+* :func:`_build_summary` — pure projection of (api_kind, record kind,
+ entity_name) onto the headline string.
+* :func:`_changed_by_dict` — projects the user columns onto the
+ ``changed_by`` DTO shape.
+"""
+
+from __future__ import annotations
+
+from typing import Any
+from uuid import UUID
+
+import sqlalchemy as sa
+
+from superset.extensions import db
+from superset.versioning.activity.impact import (
+ batch_chart_counts,
+ collect_impact_pairs,
+ impact_for_record,
+)
+from superset.versioning.activity.kinds import (
+ API_KIND_LABEL,
+ chunked_ids,
+ load_live_model,
+ NAME_COLUMN,
+ TABLE_KIND_TO_API,
+ USER_FACING_KIND,
+)
+from superset.versioning.activity.queries import check_entity_tombstones
+from superset.versioning.queries import derive_version_uuid
+
+_SUMMARY_VERBS: dict[str, str] = {
+ # The kind taxonomy mapped to past-tense verbs for the
+ # "<entity_label> <verb>: <entity_name>" headline. "field" is
+ # the fallback for scalar changes that don't map to a named verb.
+ "filter": "filter changed",
+ "metric": "metric changed",
+ "dimension": "dimension changed",
+ "column": "column changed",
+ "chart": "chart changed",
+ "time_range": "time range changed",
+ "color_palette": "palette changed",
+ "restore": "restored",
+ "field": "updated",
+}
+
+
+def apply_record_decoration(
+ records: list[dict[str, Any]],
+ path_kind: str,
+ path_id: int,
+) -> None:
+ """Add the synthesized ActivityRecord fields to each record in place:
+ ``entity_kind`` (translated to API form), ``entity_uuid``,
+ ``entity_deleted``, ``entity_deletion_state``, ``source``,
+ ``summary``, ``impact``, ``version_uuid``, ``changed_by``.
+
+ Mutates *records* in place; returns ``None``. Records are expected
+ to already carry ``entity_name`` from
+ :func:`apply_entity_name_denormalization`. The in-place mutation
+ avoids re-allocating thousands of dicts on hot dashboards; the
+ name + return signature make the side effect explicit instead of
+ pretending to be a pure projection.
+ """
+ if not records:
+ return
+
+ distinct: set[tuple[str, int]] = {
+ (
+ TABLE_KIND_TO_API.get(r["entity_kind"], ""),
+ r["entity_id"],
+ )
+ for r in records
+ if TABLE_KIND_TO_API.get(r["entity_kind"])
+ }
+ tombstones = check_entity_tombstones(distinct)
+ uuids = _lookup_entity_uuids(distinct, tombstones)
+ # Pre-compute impact counts for the whole page in one batch query
+ # instead of one COUNT per related record (was N+1).
+ impact_counts = batch_chart_counts(
+ path_id, collect_impact_pairs(records, path_kind)
+ )
+
+ for record in records:
+ api_kind = TABLE_KIND_TO_API.get(record["entity_kind"], "")
+ entity_id = record["entity_id"]
+ tombstone = tombstones.get(
+ (api_kind, entity_id), {"deleted": True, "deletion_state": None}
+ )
+ entity_uuid = uuids.get((api_kind, entity_id))
+ is_self = api_kind == path_kind and entity_id == path_id
+
+ # Emit the user-facing form ("dashboard"/"chart"/"dataset") on the
+ # wire; the internal class-name (api_kind) is kept above for the
+ # remaining decoration steps that key off model_cls.__name__.
+ record["entity_kind"] = USER_FACING_KIND.get(api_kind, api_kind)
+ record["entity_uuid"] = str(entity_uuid) if entity_uuid else None
+ record["entity_deleted"] = tombstone["deleted"]
+ record["entity_deletion_state"] = tombstone["deletion_state"]
+ record["source"] = "self" if is_self else "related"
+ record["version_uuid"] = (
+ str(derive_version_uuid(entity_uuid, record["transaction_id"]))
+ if entity_uuid
+ else None
+ )
+ record["changed_by"] = _changed_by_dict(record)
+
+ if is_self:
+ # Self records are left summary-less (the panel renders
+ # them from kind/path/values) — EXCEPT synthetic ``__meta__``
+ # headlines, whose entire payload IS the summary and whose
+ # primary surface is the entity's own stream ("restored to
+ # version N" must render on include=self).
+ record["summary"] = (
+ _build_summary(api_kind, record)
+ if record.get("kind") == "__meta__"
+ else ""
+ )
+ record["impact"] = None
+ else:
+ record["summary"] = _build_summary(api_kind, record)
+ record["impact"] = impact_for_record(record, path_kind, impact_counts)
+ if record["entity_deleted"]:
+ # Security: a tombstoned related entity has no live row, so
+ # the visibility filter cannot access-gate it (there is
+ # nothing to apply the FAB access filter to). Deletion must
+ # not WIDEN what a requester entitled only to the path entity
+ # can see, so redact everything that would identify the
+ # deleted related entity or its editors: its name, the
+ # synthesized headline, the editor identity, and the raw diff
+ # CONTENT (filter values, column names, SQL/adhoc
+ # expressions). The record still appears — kind, operation,
+ # and timestamps remain, rendered as a generic
+ # "(deleted) <kind>" marker — so the stream stays honest
+ # about WHEN something changed without disclosing WHAT, WHO,
+ # or WHICH entity. Self-path tombstones are untouched: the
+ # endpoint already gated them via ``raise_for_access`` on the
+ # path entity.
+ label = API_KIND_LABEL.get(api_kind, api_kind)
+ record["entity_name"] = ""
+ record["summary"] = f"(deleted) {label}"
+ record["changed_by"] = None
+ record["from_value"] = None
+ record["to_value"] = None
+ record["path"] = None
+
+ # Strip the internal-only columns the API contract doesn't expose.
+ for key in (
+ "entity_id",
+ "sequence",
+ "user_id",
+ "changed_by_id",
+ "first_name",
+ "last_name",
+ ):
+ record.pop(key, None)
+
+
+def _lookup_entity_uuids(
+ distinct: set[tuple[str, int]],
+ tombstones: dict[tuple[str, int], dict[str, Any]],
+) -> dict[tuple[str, int], UUID | None]:
+ """Batch-fetch live ``uuid`` per ``(api_kind, entity_id)``. Tombstoned
+ entities are skipped (their ``entity_uuid`` is null).
+ """
+ result: dict[tuple[str, int], UUID | None] = {}
+ by_kind: dict[str, list[int]] = {}
+ for api_kind, entity_id in distinct:
+ if tombstones.get((api_kind, entity_id), {}).get("deleted"):
+ continue
+ by_kind.setdefault(api_kind, []).append(entity_id)
+
+ # ``no_autoflush`` mirrors the defensive posture of the baseline +
+ # change-record listeners: this helper reads from live tables to
+ # resolve uuids, and a future caller that resolves an entity before
+ # the parent flush would otherwise trigger autoflush mid-read.
+ # Today's call sites run from request-path code with no pending
+ # session state, so the cost of the guard is zero.
+ with db.session.no_autoflush:
+ for api_kind, entity_ids in by_kind.items():
+ if api_kind not in NAME_COLUMN:
+ continue
+ model_cls = load_live_model(NAME_COLUMN[api_kind][0])
+ live_tbl = model_cls.__table__
+ # Chunk the IN-clause to stay under SQLite's bind-variable floor
+ # (a wide dashboard can have more related entities than the floor).
+ for chunk in chunked_ids(entity_ids):
+ rows = (
+ db.session.connection()
+ .execute(
+ sa.select(live_tbl.c.id, live_tbl.c.uuid).where(
+ live_tbl.c.id.in_(chunk)
+ )
+ )
+ .all()
+ )
+ for row in rows:
+ result[(api_kind, row[0])] = row[1]
Review Comment:
Addressed in the preceding review-fix commit. UUIDs are resolved from the historical shadow row at each record transaction, reused live identities are tombstoned, and related windows are clipped to the path entity incarnation boundary.
##########
superset/versioning/activity/scope.py:
##########
@@ -0,0 +1,118 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Scope resolution — turn a path entity into the related-entity walk.
+
+Composes :mod:`~superset.versioning.activity.queries` (Phase A
+relationship walks) and :mod:`~superset.versioning.activity.windows`
+(pure interval arithmetic) into the
+``list[EntityWindows]`` scope that
+:func:`~superset.versioning.activity.queries.fetch_change_records`
+consumes.
+
+The functions here read the DB (via the Phase A helpers in
+:mod:`~superset.versioning.activity.queries`); the pure window-
+arithmetic functions previously colocated here now live in
+:mod:`~superset.versioning.activity.windows` so the package no longer
+needs a lazy import to dodge a ``scope ↔ queries`` cycle.
+"""
+
+from __future__ import annotations
+
+from superset.versioning.activity.kinds import EntityWindows, Window
+from superset.versioning.activity.queries import (
+ batch_datasets_used_by_charts,
+ charts_attached_to_dashboard,
+ datasets_used_by_chart,
+)
+from superset.versioning.activity.windows import (
+ intersect_windows,
+ merge_entity_windows,
+)
+
+
+def resolve_scope(
+ path_kind: str,
+ path_id: int,
+ include: str,
+ self_start_tx: int | None = None,
+) -> list[EntityWindows]:
+ """Build the ``[(api_kind, entity_id, [windows])]`` list that
+ :func:`~superset.versioning.activity.queries.fetch_change_records`
+ consumes, branching by *path_kind* and *include* mode.
+
+ *self_start_tx* lower-bounds the self window at the path entity's first
+ tracked transaction (see
+ :func:`~superset.versioning.activity.queries.first_tracked_tx`). This
+ scopes the self stream to the current entity's own history: matching
+ self records on the bare integer id would otherwise inherit a
+ hard-deleted predecessor's change records under id reuse
+ (SQLite/MySQL reuse ``max(id)+1``). ``None`` means the entity has no
+ tracked history yet, so the self stream contributes nothing.
+ """
+ want_self = include in ("all", "self")
+ want_related = include in ("all", "related")
+
+ scope: list[EntityWindows] = []
+ if want_self and self_start_tx is not None:
+ scope.append((path_kind, path_id, [Window(self_start_tx, None)]))
+ if want_related:
+ scope.extend(_resolve_related_scope(path_kind, path_id))
+ return scope
+
+
+def _resolve_related_scope(path_kind: str, path_id: int) -> list[EntityWindows]:
+ """Walk the dependency edges from the path entity to its related
+ entities. Datasets have no transitive layer in V2."""
+ if path_kind == "Dashboard":
+ return _resolve_dashboard_scope(path_id)
+ if path_kind == "Slice":
+ return _resolve_chart_scope(path_id)
+ return []
+
+
+def _resolve_dashboard_scope(dashboard_id: int) -> list[EntityWindows]:
+ """Charts on the dashboard during their attachment window, plus
+ datasets each chart pointed at during the intersection of (chart-
+ attachment, chart-on-dataset)."""
+ scope: list[EntityWindows] = []
+ chart_windows: dict[int, list[Window]] = {}
+ for slice_id, window in charts_attached_to_dashboard(dashboard_id):
+ chart_windows.setdefault(slice_id, []).append(window)
+
+ # One query for the dataset-history of every chart on the dashboard,
+ # not one query per chart. The per-slice form was O(n_charts) round-
+ # trips which dominated p95 on rich dashboards.
+ dataset_windows_by_slice = batch_datasets_used_by_charts(set(chart_windows))
+
+ for slice_id, attachment_windows in chart_windows.items():
+ scope.append(("Slice", slice_id, list(attachment_windows)))
+ dataset_windows = dataset_windows_by_slice.get(slice_id, [])
+ for attachment in attachment_windows:
+ for dataset_id, chart_dataset_window in dataset_windows:
+ if (
+ intersect := intersect_windows(attachment, chart_dataset_window)
+ ) is not None:
+ scope.append(("SqlaTable", dataset_id, [intersect]))
Review Comment:
Addressed in the preceding review-fix commit. UUIDs are resolved from the historical shadow row at each record transaction, reused live identities are tombstoned, and related windows are clipped to the path entity incarnation boundary.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(versioning): cross-entity version activity view [superset]
Posted by "mikebridge (via GitHub)" <gi...@apache.org>.
mikebridge commented on code in PR #41076:
URL: https://github.com/apache/superset/pull/41076#discussion_r3574957700
##########
superset/commands/dashboard/importers/v1/__init__.py:
##########
@@ -201,7 +205,31 @@ def _import(
overwrite
or (dashboard.id, chart_id) not in existing_relationships
):
- dashboard_chart_ids.append((dashboard.id, chart_id))
+ target_chart_ids.append(chart_id)
+
+ if overwrite:
+ # Replace the dashboard's chart membership entirely.
+ dashboard.slices = (
+ db.session.query(Slice)
+ .filter(Slice.id.in_(target_chart_ids))
+ .all()
+ if target_chart_ids
+ else []
+ )
+ # Flush eagerly so the M2M rows land in
+ # ``dashboard_slices`` before any subsequent
+ # autoflush fires an inner-flush event handler
+ # that would reset the relationship change.
+ db.session.flush()
+ elif target_chart_ids:
+ # Append only the new associations to existing ones.
+ new_slices = (
+ db.session.query(Slice)
+ .filter(Slice.id.in_(target_chart_ids))
+ .all()
+ )
Review Comment:
Addressed by subsequent commits; the flagged code is no longer current after the rebase onto master. Resolving as outdated.
##########
superset/commands/importers/v1/utils.py:
##########
@@ -387,6 +388,37 @@ def safe_insert_dashboard_chart_relationships(
)
+def _prime_versioning_unit_of_work() -> None:
+ """Ensure Continuum has a unit-of-work for the current connection.
+
+ ``dashboard_slices`` is a Continuum-tracked (versioned) association
+ table, so a raw Core INSERT/DELETE on it fires Continuum's engine-level
+ ``before_execute`` listener, which looks up a unit-of-work for the
+ connection and raises ``KeyError`` when none is registered (the same
+ failure class the dashboard test factory hit). The normal import flow
+ registers one via prior ORM flushes, so this is belt-and-suspenders for
+ a bulk relationship insert that might run before any flush on the
+ connection. No-op (the listener is detached) when version capture is
+ disabled, which is the shipped default; never allowed to break an import.
+ """
+ try:
+ # pylint: disable=import-outside-toplevel
+ from sqlalchemy_continuum import versioning_manager
+
+ # Mirror the exact condition Continuum's track_association_operations
+ # listener uses to decide whether it acts (versioning OR
+ # native_versioning), so the prime can't skip while the listener runs.
+ options = versioning_manager.options
Review Comment:
Addressed by subsequent commits; the flagged code is no longer current after the rebase onto master. Resolving as outdated.
##########
superset/daos/dataset.py:
##########
@@ -275,6 +275,103 @@ def update(
return super().update(item, attributes)
+ @classmethod
+ def _validate_column_date_formats(
+ cls, property_columns: list[dict[str, Any]]
+ ) -> None:
+ for column in property_columns:
+ if column.get("python_date_format") is None:
+ continue
+ if not DatasetDAO.validate_python_date_format(column["python_date_format"]):
+ raise ValueError(
+ "python_date_format is an invalid date/timestamp format."
+ )
+
+ @classmethod
+ def _override_columns(
+ cls, model: SqlaTable, property_columns: list[dict[str, Any]]
+ ) -> None:
+ """Replace columns by natural key (``column_name``) — update in place
+ rather than delete-and-reinsert.
+
+ SPIKE (full-Continuum): the previous
+ delete-and-reinsert pattern produced overlapping shadow rows in
+ ``table_columns_version`` (the same ``column_name`` had a DELETE
+ shadow at tx N alongside an INSERT shadow at tx N for a fresh PK).
+ Continuum's ``Reverter`` couldn't unwind this on restore: its flush
+ ordering inserts the historical row before deleting the live one,
+ hitting the ``UNIQUE (table_id, column_name)`` constraint mid-flush
+ (ADR-004 Failure 1).
+
+ The natural-key upsert keeps PKs stable across metadata refresh.
+ Continuum captures only real field changes; new columns get plain
+ INSERT shadows; removed columns get plain DELETE shadows. No
+ natural-key collisions, so Reverter can restore cleanly.
+
+ Behaviour change vs. the previous implementation: PKs of unchanged
+ columns are preserved. Charts that reference columns by their
+ ``id`` continue to work across a metadata refresh — previously
+ such references would be invalidated.
+ """
+ existing_by_name = {c.column_name: c for c in model.columns}
Review Comment:
Addressed by subsequent commits; the flagged code is no longer current after the rebase onto master. Resolving as outdated.
##########
superset/daos/dataset.py:
##########
@@ -275,6 +275,103 @@ def update(
return super().update(item, attributes)
+ @classmethod
+ def _validate_column_date_formats(
+ cls, property_columns: list[dict[str, Any]]
+ ) -> None:
+ for column in property_columns:
+ if column.get("python_date_format") is None:
+ continue
+ if not DatasetDAO.validate_python_date_format(column["python_date_format"]):
+ raise ValueError(
+ "python_date_format is an invalid date/timestamp format."
+ )
+
+ @classmethod
+ def _override_columns(
+ cls, model: SqlaTable, property_columns: list[dict[str, Any]]
+ ) -> None:
+ """Replace columns by natural key (``column_name``) — update in place
+ rather than delete-and-reinsert.
+
+ SPIKE (full-Continuum): the previous
+ delete-and-reinsert pattern produced overlapping shadow rows in
+ ``table_columns_version`` (the same ``column_name`` had a DELETE
+ shadow at tx N alongside an INSERT shadow at tx N for a fresh PK).
+ Continuum's ``Reverter`` couldn't unwind this on restore: its flush
+ ordering inserts the historical row before deleting the live one,
+ hitting the ``UNIQUE (table_id, column_name)`` constraint mid-flush
+ (ADR-004 Failure 1).
+
+ The natural-key upsert keeps PKs stable across metadata refresh.
+ Continuum captures only real field changes; new columns get plain
+ INSERT shadows; removed columns get plain DELETE shadows. No
+ natural-key collisions, so Reverter can restore cleanly.
+
+ Behaviour change vs. the previous implementation: PKs of unchanged
+ columns are preserved. Charts that reference columns by their
+ ``id`` continue to work across a metadata refresh — previously
+ such references would be invalidated.
+ """
+ existing_by_name = {c.column_name: c for c in model.columns}
+ incoming_by_name = {p["column_name"]: p for p in property_columns}
+
+ # Identity is the natural key here, never the payload's ``id``:
+ # setattr-ing an incoming ``id`` onto a name-matched row would
+ # rewrite a live primary key, and a renamed column whose payload
+ # still carries its old ``id`` would INSERT with a live PK while
+ # the old-named row is deleted in the same flush — INSERTs flush
+ # before DELETEs, so that collides on the PK / UNIQUE(table_id,
+ # column_name) constraints. ``table_id`` is pinned to *model*.
+ protected_keys = ("id", "table_id")
Review Comment:
Addressed by subsequent commits; the flagged code is no longer current after the rebase onto master. Resolving as outdated.
##########
superset/daos/dataset.py:
##########
@@ -275,6 +275,103 @@ def update(
return super().update(item, attributes)
+ @classmethod
+ def _validate_column_date_formats(
+ cls, property_columns: list[dict[str, Any]]
+ ) -> None:
+ for column in property_columns:
+ if column.get("python_date_format") is None:
+ continue
+ if not DatasetDAO.validate_python_date_format(column["python_date_format"]):
+ raise ValueError(
+ "python_date_format is an invalid date/timestamp format."
+ )
+
+ @classmethod
+ def _override_columns(
+ cls, model: SqlaTable, property_columns: list[dict[str, Any]]
+ ) -> None:
+ """Replace columns by natural key (``column_name``) — update in place
+ rather than delete-and-reinsert.
+
+ SPIKE (full-Continuum): the previous
+ delete-and-reinsert pattern produced overlapping shadow rows in
+ ``table_columns_version`` (the same ``column_name`` had a DELETE
+ shadow at tx N alongside an INSERT shadow at tx N for a fresh PK).
+ Continuum's ``Reverter`` couldn't unwind this on restore: its flush
+ ordering inserts the historical row before deleting the live one,
+ hitting the ``UNIQUE (table_id, column_name)`` constraint mid-flush
+ (ADR-004 Failure 1).
+
+ The natural-key upsert keeps PKs stable across metadata refresh.
+ Continuum captures only real field changes; new columns get plain
+ INSERT shadows; removed columns get plain DELETE shadows. No
+ natural-key collisions, so Reverter can restore cleanly.
+
+ Behaviour change vs. the previous implementation: PKs of unchanged
+ columns are preserved. Charts that reference columns by their
+ ``id`` continue to work across a metadata refresh — previously
+ such references would be invalidated.
+ """
+ existing_by_name = {c.column_name: c for c in model.columns}
+ incoming_by_name = {p["column_name"]: p for p in property_columns}
+
+ # Identity is the natural key here, never the payload's ``id``:
+ # setattr-ing an incoming ``id`` onto a name-matched row would
+ # rewrite a live primary key, and a renamed column whose payload
+ # still carries its old ``id`` would INSERT with a live PK while
+ # the old-named row is deleted in the same flush — INSERTs flush
+ # before DELETEs, so that collides on the PK / UNIQUE(table_id,
+ # column_name) constraints. ``table_id`` is pinned to *model*.
+ protected_keys = ("id", "table_id")
+
+ # Update columns present in both: in-place setattr.
+ for name, col in existing_by_name.items():
+ if name in incoming_by_name:
+ for key, value in incoming_by_name[name].items():
+ if key not in protected_keys:
+ setattr(col, key, value)
+
+ # Insert columns present only in incoming.
+ for name, properties in incoming_by_name.items():
+ if name not in existing_by_name:
+ cleaned = {
+ key: value
+ for key, value in properties.items()
+ if key not in protected_keys
+ }
+ db.session.add(TableColumn(**{**cleaned, "table_id": model.id}))
+
+ # Delete columns present only in existing.
+ for name, col in existing_by_name.items():
+ if name not in incoming_by_name:
+ db.session.delete(col)
+
+ @classmethod
+ def _upsert_columns(
+ cls, model: SqlaTable, property_columns: list[dict[str, Any]]
+ ) -> None:
+ columns_by_id = {column.id: column for column in model.columns}
Review Comment:
Addressed by subsequent commits; the flagged code is no longer current after the rebase onto master. Resolving as outdated.
##########
superset/daos/dataset.py:
##########
@@ -363,6 +411,9 @@ def update_metrics(
- If a metric Dict does not have an `id` then we create a new metric.
- If there are extra metrics on the metadata db that are not defined on the List
then we delete.
+
+ Uses individual ORM operations (not bulk) so that SQLAlchemy-Continuum
+ can capture each row change in the version history.
"""
metrics_by_id = {metric.id: metric for metric in model.metrics}
Review Comment:
Addressed by subsequent commits; the flagged code is no longer current after the rebase onto master. Resolving as outdated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(versioning): cross-entity version activity view [superset]
Posted by "bito-code-review[bot] (via GitHub)" <gi...@apache.org>.
bito-code-review[bot] commented on PR #41076:
URL: https://github.com/apache/superset/pull/41076#issuecomment-4910677650
AI Code Review is in progress (usually takes 3 to 15 minutes unless it's a very large PR).
<details>
<summary>
Bito Usage Guide
</summary>
**Commands**
Type the following command in the pull request comment and save the comment.
- `/review` - Manually triggers a full AI review.
- `/pause` - Pauses automatic reviews on this pull request.
- `/resume` - Resumes automatic reviews.
- `/resolve` - Marks all Bito-posted review comments as resolved.
- `/abort` - Cancels all in-progress reviews.
Refer to the <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/available-commands" target="_blank" rel="noopener noreferrer">documentation</a> for additional commands.
**Configuration**
This repository uses `Superset` You can customize the agent settings <a href="https://alpha.bito.ai/home/ai-agents/code-review-agent" target="_blank" rel="noopener noreferrer">here</a> or contact your Bito workspace admin at evan@preset.io.
**Documentation & Help**
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/getting-started/install-run-using-bito-cloud/create-or-customize-an-agent-instance" target="_blank" rel="noopener noreferrer">Customize agent settings</a>
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/implementing-custom-code-review-rules" target="_blank" rel="noopener noreferrer">Review rules</a>
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent" target="_blank" rel="noopener noreferrer">General documentation</a>
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/faqs" target="_blank" rel="noopener noreferrer">FAQ</a>
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(versioning): cross-entity version activity view [superset]
Posted by "bito-code-review[bot] (via GitHub)" <gi...@apache.org>.
bito-code-review[bot] commented on PR #41076:
URL: https://github.com/apache/superset/pull/41076#issuecomment-4920016672
AI Code Review is in progress (usually takes 3 to 15 minutes unless it's a very large PR).
<details>
<summary>
Bito Usage Guide
</summary>
**Commands**
Type the following command in the pull request comment and save the comment.
- `/review` - Manually triggers a full AI review.
- `/pause` - Pauses automatic reviews on this pull request.
- `/resume` - Resumes automatic reviews.
- `/resolve` - Marks all Bito-posted review comments as resolved.
- `/abort` - Cancels all in-progress reviews.
Refer to the <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/available-commands" target="_blank" rel="noopener noreferrer">documentation</a> for additional commands.
**Configuration**
This repository uses `Superset` You can customize the agent settings <a href="https://alpha.bito.ai/home/ai-agents/code-review-agent" target="_blank" rel="noopener noreferrer">here</a> or contact your Bito workspace admin at evan@preset.io.
**Documentation & Help**
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/getting-started/install-run-using-bito-cloud/create-or-customize-an-agent-instance" target="_blank" rel="noopener noreferrer">Customize agent settings</a>
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/implementing-custom-code-review-rules" target="_blank" rel="noopener noreferrer">Review rules</a>
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent" target="_blank" rel="noopener noreferrer">General documentation</a>
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/faqs" target="_blank" rel="noopener noreferrer">FAQ</a>
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(versioning): cross-entity version activity view [superset]
Posted by "codeant-ai-for-open-source[bot] (via GitHub)" <gi...@apache.org>.
codeant-ai-for-open-source[bot] commented on code in PR #41076:
URL: https://github.com/apache/superset/pull/41076#discussion_r3560236305
##########
tests/integration_tests/versioning/activity_view_tests.py:
##########
@@ -0,0 +1,1250 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Integration tests for the cross-entity activity-view API (sc-107283).
+
+US1 — dashboard activity stream: ``GET /api/v1/dashboard/<uuid>/activity/``.
+Tests for US2 (chart activity) and US3 (dataset activity) come in later
+phases.
+
+Per spec T053 / sc-103156 T062, every test that mutates a fixture entity
+wraps the test body in ``try``/``finally`` with
+``metadata_db.session.rollback()`` in the ``finally``. The rationale is
+documented in the spec — Continuum captures dirty mappers during
+autoflush, so leaving an instrumented attribute dirty pollutes
+downstream tests via the shadow tables.
+"""
+
+from __future__ import annotations
+
+from typing import Any
+from uuid import uuid4
+
+import pytest
+
+from superset.connectors.sqla.models import SqlaTable
+from superset.extensions import db
+from superset.models.dashboard import Dashboard
+from superset.models.slice import Slice
+from superset.utils import json as _json
+from tests.integration_tests.base_tests import SupersetTestCase
+from tests.integration_tests.constants import (
+ ADMIN_USERNAME,
+ ALPHA_USERNAME,
+ GAMMA_USERNAME,
+)
+from tests.integration_tests.fixtures.birth_names_dashboard import ( # noqa: F401
+ load_birth_names_dashboard_with_slices,
+ load_birth_names_data,
+)
+
+
+def _get_birth_names_dataset() -> SqlaTable:
+ return (
+ db.session.query(SqlaTable)
+ .filter(SqlaTable.table_name == "birth_names")
+ .first()
+ )
+
+
+def _persist_fixture_state() -> None:
+ """Force the fixture's pending INSERTs to commit so subsequent edits
+ produce *new* version rows instead of being batched into the
+ creation transaction. Mirrors the same helper in
+ ``tests/integration_tests/dashboards/version_history_tests.py``.
+ """
+ db.session.commit()
+
+
+def _get_birth_names_dashboard() -> Dashboard:
+ return (
+ db.session.query(Dashboard)
+ .filter(Dashboard.dashboard_title == "USA Births Names")
+ .first()
+ )
+
+
+class TestDashboardActivityView(SupersetTestCase):
+ """T017–T026 — ``GET /api/v1/dashboard/<uuid>/activity/`` (US1)."""
+
+ @pytest.fixture(autouse=True)
+ def _load_data(self, load_birth_names_dashboard_with_slices): # noqa: PT004, F811
+ pass
+
+ def _activity(self, dashboard_uuid: str, **query: Any) -> Any:
+ return self.client.get(
+ f"/api/v1/dashboard/{dashboard_uuid}/activity/",
+ query_string=query,
+ )
+
+ # ---- 4xx boundary cases ----
+
+ def test_activity_returns_404_for_unknown_uuid(self) -> None:
+ """AV-009: unknown path entity → 404."""
+ self.login(ADMIN_USERNAME)
+ rv = self._activity("00000000-0000-0000-0000-000000000000")
+ assert rv.status_code == 404
+
+ def test_activity_returns_400_for_invalid_uuid(self) -> None:
+ """A malformed UUID is rejected by the endpoint, not by Werkzeug."""
+ self.login(ADMIN_USERNAME)
+ rv = self._activity("not-a-uuid")
+ assert rv.status_code == 400
+
+ def test_activity_returns_400_for_invalid_include(self) -> None:
+ _persist_fixture_state()
+ dashboard = _get_birth_names_dashboard()
+ assert dashboard is not None
+ self.login(ADMIN_USERNAME)
+ rv = self._activity(str(dashboard.uuid), include="sibling")
+ assert rv.status_code == 400
+
+ def test_activity_returns_400_for_invalid_since(self) -> None:
+ _persist_fixture_state()
+ dashboard = _get_birth_names_dashboard()
+ assert dashboard is not None
+ self.login(ADMIN_USERNAME)
+ rv = self._activity(str(dashboard.uuid), since="yesterday")
+ assert rv.status_code == 400
+
+ def test_activity_allows_read_non_owner(self) -> None:
+ """Activity is a read endpoint: a non-owner with read access (Alpha,
+ which carries broad read + datasource access) can read a dashboard's
+ activity stream — ``raise_for_access(dashboard=)`` does not reject —
+ so the endpoint returns 200. Visibility of *related* rows is filtered
+ separately, inside the activity layer."""
+ _persist_fixture_state()
+ dashboard = _get_birth_names_dashboard()
+ assert dashboard is not None
+ dashboard_uuid = str(dashboard.uuid)
+
+ self.login(ALPHA_USERNAME)
+ rv = self._activity(dashboard_uuid)
+ assert rv.status_code == 200
+
+ def test_visibility_filter_silently_drops_inaccessible_related(self) -> None:
+ """AV-008 security control: a related record whose entity the caller
+ cannot read is *silently* dropped — absent from the result, no
+ placeholder, no contribution to count. Exercises the real
+ enforcement point (``filter_records_by_visibility`` →
+ ``DashboardAccessFilter``) with a restricted Gamma principal.
+
+ Setup uses two dashboards (no datasource needed): one owned by
+ Gamma (readable), and an admin-owned one Gamma may not read.
+ """
+ from superset import security_manager
+ from superset.subjects.utils import get_user_subject
+ from superset.versioning.activity.visibility import (
+ filter_records_by_visibility,
+ )
+
+ admin = security_manager.find_user(ADMIN_USERNAME)
+ gamma = security_manager.find_user(GAMMA_USERNAME)
+ # Access control is subject-based (``editors``) since the Subject
+ # work replaced ``owners``; grant read via each user's Subject.
+ visible = Dashboard(
+ dashboard_title=f"vis-probe-owned {uuid4().hex[:8]}",
+ slug=f"vis-owned-{uuid4().hex[:8]}",
+ published=False,
+ editors=[get_user_subject(gamma.id)],
+ )
+ hidden = Dashboard(
+ dashboard_title=f"vis-probe-hidden {uuid4().hex[:8]}",
+ slug=f"vis-hidden-{uuid4().hex[:8]}",
+ published=False,
+ editors=[get_user_subject(admin.id)],
+ )
+ db.session.add_all([visible, hidden])
+ db.session.commit()
+ visible_id, hidden_id = visible.id, hidden.id
+ try:
+ records = [
+ {"entity_kind": "dashboard", "entity_id": visible_id},
+ {"entity_kind": "dashboard", "entity_id": hidden_id},
+ ]
+ self.login(GAMMA_USERNAME)
+ filtered = filter_records_by_visibility(records)
+ kept_ids = {r["entity_id"] for r in filtered}
+
+ assert visible_id in kept_ids, (
+ "gamma-owned dashboard must stay visible to Gamma"
+ )
+ assert hidden_id not in kept_ids, (
+ "unpublished admin-owned dashboard must be dropped for Gamma"
+ )
+ # Silent: the dropped record leaves nothing behind (no placeholder).
+ assert len(filtered) == 1
+ finally:
+ db.session.rollback()
+ for did in (visible_id, hidden_id):
+ obj = db.session.query(Dashboard).filter(Dashboard.id == did).first()
+ if obj is not None:
+ db.session.delete(obj)
+ db.session.commit()
+
+ # ---- 200 happy paths ----
+
+ def test_activity_returns_200_with_envelope_shape(self) -> None:
+ """Smoke test: the endpoint returns the documented envelope shape
+ (``result`` list + ``count`` integer) even when the dashboard has
+ no activity yet."""
+ _persist_fixture_state()
+ dashboard = _get_birth_names_dashboard()
+ assert dashboard is not None
+ dashboard_uuid = str(dashboard.uuid)
+
+ self.login(ADMIN_USERNAME)
+ rv = self._activity(dashboard_uuid)
+ assert rv.status_code == 200
+ body = _json.loads(rv.data.decode("utf-8"))
+ assert "result" in body
+ assert "count" in body
+ assert isinstance(body["result"], list)
+ assert isinstance(body["count"], int)
+
+ def test_activity_includes_chart_edit_as_related(self) -> None:
+ """T018 / AS-1 of US1: editing a chart on the dashboard surfaces
+ the chart-edit record with ``entity_kind=Slice`` and
+ ``source=related``."""
+ _persist_fixture_state()
+ dashboard = _get_birth_names_dashboard()
+ assert dashboard is not None
+ dashboard_uuid = str(dashboard.uuid)
+ chart_on_dashboard = next(iter(dashboard.slices), None)
+ assert chart_on_dashboard is not None
+ chart_id = chart_on_dashboard.id
+ original_name = chart_on_dashboard.slice_name
+
+ try:
+ chart_on_dashboard.slice_name = f"{original_name} (edited)"
+ db.session.commit()
+
+ self.login(ADMIN_USERNAME)
+ rv = self._activity(dashboard_uuid)
+ assert rv.status_code == 200
+ body = _json.loads(rv.data.decode("utf-8"))
+ related = [
+ r
+ for r in body["result"]
+ if r["entity_kind"] == "chart" and r["source"] == "related"
+ ]
+ assert related, (
+ "Expected at least one Slice/related record from the chart "
+ "edit; got: "
+ f"{[(r['entity_kind'], r['source']) for r in body['result']]}"
+ )
+ # Spot-check the carry-through of denormalized fields
+ sample = related[0]
+ assert sample["entity_uuid"] is not None
+ assert "transaction_id" in sample
+ assert "issued_at" in sample
+ finally:
+ db.session.rollback()
+ chart = db.session.query(Slice).filter(Slice.id == chart_id).one()
+ chart.slice_name = original_name
+ db.session.commit()
+
+ def test_activity_include_self_excludes_related(self) -> None:
+ """T023 / AV-016: ``?include=self`` filters out related records."""
+ _persist_fixture_state()
+ dashboard = _get_birth_names_dashboard()
+ assert dashboard is not None
+ dashboard_uuid = str(dashboard.uuid)
+ chart_on_dashboard = next(iter(dashboard.slices), None)
+ assert chart_on_dashboard is not None
+ chart_id = chart_on_dashboard.id
+ original_name = chart_on_dashboard.slice_name
+
+ try:
+ chart_on_dashboard.slice_name = f"{original_name} (edited self)"
+ db.session.commit()
+
+ self.login(ADMIN_USERNAME)
+ rv = self._activity(dashboard_uuid, include="self")
+ assert rv.status_code == 200
+ body = _json.loads(rv.data.decode("utf-8"))
+ for record in body["result"]:
+ assert record["source"] == "self", (
+ f"include=self leaked a non-self record: {record}"
+ )
+ assert record["entity_kind"] == "dashboard"
+ finally:
+ db.session.rollback()
+ chart = db.session.query(Slice).filter(Slice.id == chart_id).one()
+ chart.slice_name = original_name
+ db.session.commit()
+
+ def test_activity_include_related_excludes_self(self) -> None:
+ """T024 / AV-016: ``?include=related`` returns only related records."""
+ _persist_fixture_state()
+ dashboard = _get_birth_names_dashboard()
+ assert dashboard is not None
+ dashboard_uuid = str(dashboard.uuid)
+ original_title = dashboard.dashboard_title
+ dashboard_id = dashboard.id
+
+ try:
+ # Edit the dashboard's own field so we have a self record to
+ # filter out, and edit a chart on it so we have a related
+ # record to keep.
+ dashboard.dashboard_title = f"{original_title} (edited dash)"
+ db.session.commit()
+ chart_on_dashboard = next(iter(dashboard.slices), None)
+ assert chart_on_dashboard is not None
+ chart_id = chart_on_dashboard.id
+ chart_original_name = chart_on_dashboard.slice_name
+ chart_on_dashboard.slice_name = f"{chart_original_name} (edited chart)"
+ db.session.commit()
+
+ self.login(ADMIN_USERNAME)
+ rv = self._activity(dashboard_uuid, include="related")
+ assert rv.status_code == 200
+ body = _json.loads(rv.data.decode("utf-8"))
+ for record in body["result"]:
+ assert record["source"] == "related", (
+ f"include=related leaked a self record: {record}"
+ )
+ assert record["entity_kind"] != "dashboard"
+ finally:
+ db.session.rollback()
+ dashboard = (
+ db.session.query(Dashboard).filter(Dashboard.id == dashboard_id).one()
+ )
+ dashboard.dashboard_title = original_title
+ chart = db.session.query(Slice).filter(Slice.id == chart_id).one()
+ chart.slice_name = chart_original_name
+ db.session.commit()
Review Comment:
**Suggestion:** `chart_id` and `chart_original_name` are created inside the `try` block but used unconditionally in `finally`; if an earlier assertion/exception fires before assignment, cleanup raises `UnboundLocalError` and masks the real failure. Initialize these variables before `try` and guard the chart-restore cleanup when they are unavailable. [incorrect variable usage]
<details>
<summary><b>Severity Level:</b> Major ⚠️</summary>
```mdx
- ⚠️ Activity-view dashboard test failures masked by cleanup UnboundLocalError.
- ⚠️ Harder to diagnose fixture or data regressions in tests.
```
</details>
<details>
<summary><b>Steps of Reproduction ✅ </b></summary>
```mdx
1. Run the dashboard activity tests, e.g. `pytest
tests/integration_tests/versioning/activity_view_tests.py::TestDashboardActivityView::test_activity_include_related_excludes_self`,
which exercises the block starting at line 290 in
`tests/integration_tests/versioning/activity_view_tests.py`.
2. Inside `test_activity_include_related_excludes_self`, note that `chart_id` and
`chart_original_name` are first assigned inside the `try:` block after the assertion at
lines 305–307:
- line 305: `chart_on_dashboard = next(iter(dashboard.slices), None)`
- line 306: `assert chart_on_dashboard is not None`
- line 307: `chart_id = chart_on_dashboard.id`
- line 308: `chart_original_name = chart_on_dashboard.slice_name`
3. If the fixture or database state is ever such that `dashboard.slices` is empty (for
example, the birth_names dashboard is created without slices or prior tests/fixture
changesfail to attach any charts), the assertion at line 306 fails before `chart_id` and
`chart_original_name` are assigned.
4. When that assertion (or any earlier exception in the `try:` block before line 307)
fires, Python immediately enters the `finally:` block at lines 321–329, which
unconditionally executes:
- line 327: `chart = db.session.query(Slice).filter(Slice.id == chart_id).one()`
- line 328: `chart.slice_name = chart_original_name`
Because `chart_id` and/or `chart_original_name` were never bound, this raises
`UnboundLocalError` in the cleanup, masking the original failure that actually caused
the test to abort.
```
</details>
[](https://app.codeant.ai/fix-in-ide?tool=cursor&prompt_id=5110c51fecc9482cbf11b28589ac3691&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset) [](https://app.codeant.ai/fix-in-ide?tool=vscode-claude&prompt_id=5110c51fecc9482cbf11b28589ac3691&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset)
*(Use Cmd/Ctrl + Click for best experience)*
<details>
<summary><b>Prompt for AI Agent 🤖 </b></summary>
```mdx
This is a comment left during a code review.
**Path:** tests/integration_tests/versioning/activity_view_tests.py
**Line:** 321:329
**Comment:**
*Incorrect Variable Usage: `chart_id` and `chart_original_name` are created inside the `try` block but used unconditionally in `finally`; if an earlier assertion/exception fires before assignment, cleanup raises `UnboundLocalError` and masks the real failure. Initialize these variables before `try` and guard the chart-restore cleanup when they are unavailable.
Validate the correctness of the flagged issue. If correct, How can I resolve this? If you propose a fix, implement it and please make it concise.
Once fix is implemented, also check other comments on the same PR, and ask user if the user wants to fix the rest of the comments as well. if said yes, then fetch all the comments validate the correctness and implement a minimal fix
```
</details>
<a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F41076&comment_hash=f2cc2cf06cca0430b8bd6ae4c5b49f884ac639a0cf789daacbabfcd02c0060ac&reaction=like'>👍</a> | <a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F41076&comment_hash=f2cc2cf06cca0430b8bd6ae4c5b49f884ac639a0cf789daacbabfcd02c0060ac&reaction=dislike'>👎</a>
##########
tests/integration_tests/versioning/perf_validation_tests.py:
##########
@@ -0,0 +1,416 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""T044 — Performance validation for entity version history.
+
+Skipped by default. Run on demand:
+
+ SUPERSET_PERF_VALIDATION=1 pytest \
+ tests/integration_tests/versioning/perf_validation_tests.py -v -s
+
+Measures the three success criteria defined in the spec:
+
+ * SC-002: version list endpoint responds in under 1 second
+ * SC-004: save path p95 overhead under 50 ms with Continuum tracking
+ on vs. off (FR-014)
+
+(SC-003, the restore-endpoint benchmark, lands with the restore endpoint
+in a later PR.)
+
+The test prints a summary table suitable for pasting into the PR
+description. It also asserts each target so regressions fail loudly
+when the harness is re-run.
+"""
+
+from __future__ import annotations
+
+import os
+import statistics
+import time
+from typing import Any
+
+import pytest
+import sqlalchemy as sa
+from sqlalchemy_continuum import version_class, versioning_manager
+
+from superset.extensions import db
+from superset.models.slice import Slice
+from tests.integration_tests.base_tests import SupersetTestCase
+from tests.integration_tests.constants import ADMIN_USERNAME
+from tests.integration_tests.fixtures.birth_names_dashboard import ( # noqa: F401
+ load_birth_names_dashboard_with_slices,
+ load_birth_names_data,
+)
+
+SKIP_REASON = "Performance validation is manual. Set SUPERSET_PERF_VALIDATION=1 to run."
+
+# Thresholds from spec.md §Success Criteria.
+LIST_ENDPOINT_MAX_MS = 1000 # SC-002
+SAVE_OVERHEAD_P95_MAX_MS = 50 # SC-004
+
+# Activity-view thresholds (sc-107283 §Success Criteria).
+ACTIVITY_ENDPOINT_P95_MAX_MS = 1500 # SC-AV-001
+
+
+def _save_chart_once(chart: Slice, suffix: str) -> None:
+ """One ORM-level save path, mimicking what ChartDAO.update does."""
+ chart.slice_name = f"{chart.slice_name[:64]}_{suffix}"
+ db.session.commit()
+
+
+def _timings_ms(seconds: list[float]) -> dict[str, float]:
+ ms = sorted(s * 1000.0 for s in seconds)
+ return {
+ "p50": statistics.median(ms),
+ "p95": ms[int(len(ms) * 0.95) - 1] if len(ms) >= 20 else max(ms),
+ "max": max(ms),
+ "n": len(ms),
+ }
+
+
+@pytest.mark.skipif(
+ not os.environ.get("SUPERSET_PERF_VALIDATION"),
+ reason=SKIP_REASON,
+)
+class PerfValidationTests(SupersetTestCase):
+ """Runs only when SUPERSET_PERF_VALIDATION=1 is set."""
+
+ @pytest.fixture(autouse=True)
+ def _load_data(self, load_birth_names_dashboard_with_slices: Any) -> None: # noqa: F811, PT004
+ pass
+
+ def _seed_chart_with_n_versions(self, n: int) -> Slice:
+ """Save a chart N times to produce N version rows."""
+ chart = db.session.query(Slice).first()
+ assert chart is not None, "birth_names fixture should provide charts"
+
+ for i in range(n):
+ _save_chart_once(chart, f"v{i}")
+ db.session.commit()
+ return chart
+
+ def test_sc002_list_endpoint_under_1s(self) -> None:
+ """SC-002: list endpoint responds in under 1 second."""
+ self.login(ADMIN_USERNAME)
+
+ # Generate enough versions to exercise the retention-capped state.
+ chart = self._seed_chart_with_n_versions(24)
+ chart_uuid = str(chart.uuid)
+ url = f"/api/v1/chart/{chart_uuid}/versions/"
+
+ # Warm up the endpoint once (JIT caching, mapper configuration, etc.)
+ self.client.get(url)
+
+ timings: list[float] = []
+ for _ in range(10):
+ t0 = time.perf_counter()
+ response = self.client.get(url)
+ timings.append(time.perf_counter() - t0)
+ assert response.status_code == 200
+
+ stats = _timings_ms(timings)
+ print(
+ f"\n[SC-002] GET /versions/ (24 versions) "
+ f"p50={stats['p50']:.1f}ms p95={stats['p95']:.1f}ms "
+ f"max={stats['max']:.1f}ms n={stats['n']}"
+ )
+ assert stats["p95"] < LIST_ENDPOINT_MAX_MS, (
+ f"SC-002 failed: list endpoint p95 {stats['p95']:.1f}ms "
+ f">= {LIST_ENDPOINT_MAX_MS}ms"
+ )
+
+ # SC-003 (restore endpoint under 3s) intentionally omitted here: the
+ # version-restore route (POST /<uuid>/versions/<version_uuid>/restore)
+ # ships in a later PR, so there is nothing to time at this point in the
+ # stack. Re-add this benchmark alongside the restore endpoint.
+
+ def test_sc004_save_overhead_under_50ms(self) -> None:
+ """SC-004: save path p95 overhead under 50ms (FR-014).
+
+ Toggling Continuum on and off mid-process corrupts its internal
+ ``units_of_work`` state and is not a reliable measurement. Instead
+ this test directly measures the wall-clock time spent inside the
+ four session-level listeners Continuum attaches to
+ ``sa.orm.session.Session`` — ``before_flush``, ``after_flush``,
+ ``after_commit``, ``after_rollback`` — plus Superset's own
+ baseline / snapshot / retention-prune listeners (attached to
+ ``db.session``). The cumulative listener time per save is the
+ marginal overhead version capture adds over a save with
+ versioning removed entirely, because without these listeners
+ the ORM would not execute any of that code.
+
+ The approach:
+ 1. Wrap each known listener with a timing proxy that adds its
+ wall-clock time to a per-save accumulator.
+ 2. Save the same chart N times, recording each save's
+ accumulator value.
+ 3. Compute p50 / p95 of the per-save overhead.
+
+ This matches the measurement intent of SC-004 (how much does
+ versioning cost per save) without the fragility of toggling
+ Continuum mid-test.
+ """
+ self.login(ADMIN_USERNAME)
+
+ chart = db.session.query(Slice).first()
+ assert chart is not None
+
+ # Per-save accumulator incremented by the wrapped listeners.
+ acc = [0.0]
+
+ def wrap_listener(original: Any) -> Any:
+ def wrapper(*args: Any, **kwargs: Any) -> Any:
+ t0 = time.perf_counter()
+ try:
+ return original(*args, **kwargs)
+ finally:
+ acc[0] += time.perf_counter() - t0
+
+ wrapper.__wrapped__ = original # type: ignore[attr-defined]
+ return wrapper
+
+ # Instrument Continuum's four session listeners by detaching the
+ # bound method, wrapping, and re-attaching under a single-use
+ # listener handle we can cleanly remove on teardown.
+ session_target = sa.orm.session.Session
+ attached: list[tuple[str, Any]] = []
+ for event_name, listener in list(versioning_manager.session_listeners.items()):
+ sa.event.remove(session_target, event_name, listener)
+ wrapped = wrap_listener(listener)
+ sa.event.listen(session_target, event_name, wrapped)
+ attached.append((event_name, wrapped))
Review Comment:
**Suggestion:** The overhead meter wraps only `versioning_manager.session_listeners` on `sa.orm.session.Session`, but the Superset versioning listeners being benchmarked are registered on `db.session`. This undercounts save-path overhead and can produce false passes for SC-004/AV-SC-003. Wrap and time the `db.session` listeners as well (or switch to a measurement point that includes both listener groups) so the benchmark reflects real save overhead. [logic error]
<details>
<summary><b>Severity Level:</b> Major ⚠️</summary>
```mdx
- ⚠️ Perf harness underestimates save-path overhead from Superset listeners.
- ⚠️ Performance regressions may slip past SC-004/AV-SC-003 checks.
```
</details>
<details>
<summary><b>Steps of Reproduction ✅ </b></summary>
```mdx
1. Enable the perf harness by setting `SUPERSET_PERF_VALIDATION=1` and run `pytest
tests/integration_tests/versioning/perf_validation_tests.py::PerfValidationTests::test_sc004_save_overhead_under_50ms`
(and similarly `::test_av_sc003_save_path_p95_unaffected_by_activity_view`), which execute
the listener-wrapping code at lines 185–195 and 376–382 in
`tests/integration_tests/versioning/perf_validation_tests.py`.
2. In `test_sc004_save_overhead_under_50ms`, observe the instrumentation block at lines
188–194:
- line 188: `session_target = sa.orm.session.Session`
- line 190: loop over `versioning_manager.session_listeners.items()`
- lines 191–193: remove and re-attach each listener on `session_target` wrapped with
`wrap_listener`, accumulating per-save overhead in `acc[0]`.
This only wraps the Continuum listeners stored in
`versioning_manager.session_listeners` on the `Session` class.
3. Separately, inspect `superset/versioning/changes/listener.py` at lines 428–542, where
Superset’s own versioning listeners are registered directly on `db.session` (a scoped
session), not on the `Session` class:
- line 428: `def compute_change_records(session: Session, ...)`
- line 521+: `event.listen(db.session, "before_flush", compute_change_records)`
- line 522+: `event.listen(db.session, "after_flush", flush_change_records)`
- line 523+: `event.listen(db.session, "after_commit", reset_processed_after_commit)`
- line 524+: `event.listen(db.session, "after_rollback",
reset_action_kind_after_rollback)`
These functions run on every save and contribute to versioning overhead but are not
part of `versioning_manager.session_listeners`.
4. When the perf tests run, both the Continuum listeners (wrapped and timed) and the
Superset `db.session` listeners (unwrapped) execute on each commit. However, the
accumulator `acc[0]` in `test_sc004_save_overhead_under_50ms` and
`test_av_sc003_save_path_p95_unaffected_by_activity_view` only includes time spent in the
wrapped `versioning_manager.session_listeners` attached to `sa.orm.session.Session`. Time
spent in Superset’s `db.session` listeners (compute_change_records, flush_change_records,
etc.) is completely omitted, so the reported p50/p95 “overhead” stats can be significantly
lower than the true save-path overhead of the full versioning stack.
```
</details>
[](https://app.codeant.ai/fix-in-ide?tool=cursor&prompt_id=984817fb15114bb3b789b3baa179f5ed&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset) [](https://app.codeant.ai/fix-in-ide?tool=vscode-claude&prompt_id=984817fb15114bb3b789b3baa179f5ed&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset)
*(Use Cmd/Ctrl + Click for best experience)*
<details>
<summary><b>Prompt for AI Agent 🤖 </b></summary>
```mdx
This is a comment left during a code review.
**Path:** tests/integration_tests/versioning/perf_validation_tests.py
**Line:** 188:194
**Comment:**
*Logic Error: The overhead meter wraps only `versioning_manager.session_listeners` on `sa.orm.session.Session`, but the Superset versioning listeners being benchmarked are registered on `db.session`. This undercounts save-path overhead and can produce false passes for SC-004/AV-SC-003. Wrap and time the `db.session` listeners as well (or switch to a measurement point that includes both listener groups) so the benchmark reflects real save overhead.
Validate the correctness of the flagged issue. If correct, How can I resolve this? If you propose a fix, implement it and please make it concise.
Once fix is implemented, also check other comments on the same PR, and ask user if the user wants to fix the rest of the comments as well. if said yes, then fetch all the comments validate the correctness and implement a minimal fix
```
</details>
<a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F41076&comment_hash=ef0f646ea86c021ac275919180693ab982a7d59728d6f86e89dc3eca0a241355&reaction=like'>👍</a> | <a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F41076&comment_hash=ef0f646ea86c021ac275919180693ab982a7d59728d6f86e89dc3eca0a241355&reaction=dislike'>👎</a>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(versioning): cross-entity version activity view [superset]
Posted by "mikebridge (via GitHub)" <gi...@apache.org>.
mikebridge commented on code in PR #41076:
URL: https://github.com/apache/superset/pull/41076#discussion_r3574996320
##########
superset/migrations/versions/2026-05-28_19-50_56cd24c07170_add_versioning_tables.py:
##########
@@ -0,0 +1,562 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""add_versioning_tables
+
+Creates the full schema backing entity versioning in a single
+migration:
+
+1. ``version_transaction`` — audit log keyed by Continuum's per-flush
+ transaction id (plus a Postgres-specific id sequence).
+2. **Parent shadow tables** mirroring each versioned entity's columns:
+ ``dashboards_version`` / ``slices_version`` / ``tables_version``.
+3. ``version_changes`` — field-level diff log keyed to a
+ ``(transaction, entity)`` pair; each row describes one atomic change
+ (one field or one child-collection element) that occurred during a
+ save.
+4. **Child shadow tables** for the collections Continuum auto-registers
+ when ``__versioned__`` is applied to ``TableColumn`` / ``SqlMetric``
+ and the ``slices`` exclude is removed from
+ ``Dashboard.__versioned__``: ``table_columns_version`` /
+ ``sql_metrics_version`` / ``dashboard_slices_version``.
+
+All shadow tables follow the validity-strategy shape (mirrored columns
++ ``transaction_id`` / ``end_transaction_id`` / ``operation_type``
+bookkeeping with FKs to ``version_transaction.id``). The current
+version row has ``end_transaction_id = NULL``.
+
+This migration replaces three iterative migrations from the spike phase
+(``56cd24c07170``, ``e1f3c5a7b9d0``, ``f7a2b3c4d5e6``) that captured the
+same schema in three steps as the feature was developed. Compacting
+gives downstream operators one migration to apply / reverse and one
+review surface. The ``revision`` hash is reused from the original first
+migration so anyone still tracking the chain by that hash lands on the
+same logical change set.
+
+Generated by hand because the current Continuum + Alembic-autogenerate
+interaction trips on the renamed ``transaction`` -> ``version_transaction``
+table key (``KeyError`` lookups in ``table_key_to_table``). Column
+inventories were sourced from the live model ``__table__`` definitions
+and ``version_class(...).__table__`` / Continuum association metadata.
+
+Primary key choice. Both ``version_transaction.id`` and
+``version_changes.id`` are ``BigInteger`` autoincrement — a deliberate
+carveout from the project's UUID-PK convention for new models (see
+``CLAUDE.md`` §"UUID Migration"). ``version_transaction`` is keyed
+externally by SQLAlchemy-Continuum via
+``nextval('version_transaction_id_seq')`` on every INSERT; matching
+that contract is required for ``versioning_manager`` to function.
+``version_changes`` follows the same shape because the user-facing
+identity is the ``(transaction_id, entity_kind, entity_id, sequence)``
+composite unique key, not the row id; the API surfaces a deterministic
+UUIDv5 ``version_uuid`` derived from ``entity.uuid`` and
+``transaction_id`` for stable external references.
+
+Revision ID: 56cd24c07170
+Revises: 2bee73611e32
+Create Date: 2026-05-28 19:50:00.000000
+
+"""
+
+from __future__ import annotations
+
+import sqlalchemy as sa
+from alembic import op
+from sqlalchemy_utils import UUIDType
+
+from superset.utils.core import MediumText
+
+revision = "56cd24c07170"
+# Stacked on the composite-PK association-tables change (2bee73611e32) so the
+# Continuum shadow tables this migration creates can mirror the
+# composite-PK shape of the live association tables. If that change
+# is removed from the stack, this should be reverted to "ce6bd21901ab".
+down_revision = "2bee73611e32"
+
+
+def upgrade() -> None:
+ bind = op.get_bind()
+
+ # ------------------------------------------------------------------
+ # version_transaction
+ #
+ # Audit log for each versioning event. Continuum emits
+ # ``nextval('version_transaction_id_seq')`` on every INSERT, so the
+ # sequence must exist before the table on Postgres. SQLite/MySQL
+ # ignore the explicit CREATE SEQUENCE (they auto-increment natively).
+ # ------------------------------------------------------------------
+ if bind.dialect.name == "postgresql":
+ op.execute("CREATE SEQUENCE IF NOT EXISTS version_transaction_id_seq")
+
+ op.create_table(
Review Comment:
Addressed by subsequent commits; the flagged code is no longer current after the rebase onto master. Resolving as outdated.
##########
superset/migrations/versions/2026-05-28_19-50_56cd24c07170_add_versioning_tables.py:
##########
@@ -0,0 +1,562 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""add_versioning_tables
+
+Creates the full schema backing entity versioning in a single
+migration:
+
+1. ``version_transaction`` — audit log keyed by Continuum's per-flush
+ transaction id (plus a Postgres-specific id sequence).
+2. **Parent shadow tables** mirroring each versioned entity's columns:
+ ``dashboards_version`` / ``slices_version`` / ``tables_version``.
+3. ``version_changes`` — field-level diff log keyed to a
+ ``(transaction, entity)`` pair; each row describes one atomic change
+ (one field or one child-collection element) that occurred during a
+ save.
+4. **Child shadow tables** for the collections Continuum auto-registers
+ when ``__versioned__`` is applied to ``TableColumn`` / ``SqlMetric``
+ and the ``slices`` exclude is removed from
+ ``Dashboard.__versioned__``: ``table_columns_version`` /
+ ``sql_metrics_version`` / ``dashboard_slices_version``.
+
+All shadow tables follow the validity-strategy shape (mirrored columns
++ ``transaction_id`` / ``end_transaction_id`` / ``operation_type``
+bookkeeping with FKs to ``version_transaction.id``). The current
+version row has ``end_transaction_id = NULL``.
+
+This migration replaces three iterative migrations from the spike phase
+(``56cd24c07170``, ``e1f3c5a7b9d0``, ``f7a2b3c4d5e6``) that captured the
+same schema in three steps as the feature was developed. Compacting
+gives downstream operators one migration to apply / reverse and one
+review surface. The ``revision`` hash is reused from the original first
+migration so anyone still tracking the chain by that hash lands on the
+same logical change set.
+
+Generated by hand because the current Continuum + Alembic-autogenerate
+interaction trips on the renamed ``transaction`` -> ``version_transaction``
+table key (``KeyError`` lookups in ``table_key_to_table``). Column
+inventories were sourced from the live model ``__table__`` definitions
+and ``version_class(...).__table__`` / Continuum association metadata.
+
+Primary key choice. Both ``version_transaction.id`` and
+``version_changes.id`` are ``BigInteger`` autoincrement — a deliberate
+carveout from the project's UUID-PK convention for new models (see
+``CLAUDE.md`` §"UUID Migration"). ``version_transaction`` is keyed
+externally by SQLAlchemy-Continuum via
+``nextval('version_transaction_id_seq')`` on every INSERT; matching
+that contract is required for ``versioning_manager`` to function.
+``version_changes`` follows the same shape because the user-facing
+identity is the ``(transaction_id, entity_kind, entity_id, sequence)``
+composite unique key, not the row id; the API surfaces a deterministic
+UUIDv5 ``version_uuid`` derived from ``entity.uuid`` and
+``transaction_id`` for stable external references.
+
+Revision ID: 56cd24c07170
+Revises: 2bee73611e32
+Create Date: 2026-05-28 19:50:00.000000
+
+"""
+
+from __future__ import annotations
+
+import sqlalchemy as sa
+from alembic import op
+from sqlalchemy_utils import UUIDType
+
+from superset.utils.core import MediumText
+
+revision = "56cd24c07170"
+# Stacked on the composite-PK association-tables change (2bee73611e32) so the
+# Continuum shadow tables this migration creates can mirror the
+# composite-PK shape of the live association tables. If that change
+# is removed from the stack, this should be reverted to "ce6bd21901ab".
+down_revision = "2bee73611e32"
+
+
+def upgrade() -> None:
+ bind = op.get_bind()
+
+ # ------------------------------------------------------------------
+ # version_transaction
+ #
+ # Audit log for each versioning event. Continuum emits
+ # ``nextval('version_transaction_id_seq')`` on every INSERT, so the
+ # sequence must exist before the table on Postgres. SQLite/MySQL
+ # ignore the explicit CREATE SEQUENCE (they auto-increment natively).
+ # ------------------------------------------------------------------
+ if bind.dialect.name == "postgresql":
+ op.execute("CREATE SEQUENCE IF NOT EXISTS version_transaction_id_seq")
+
+ op.create_table(
+ "version_transaction",
+ sa.Column(
+ "id",
+ sa.BigInteger(),
+ sa.Sequence("version_transaction_id_seq"),
+ primary_key=True,
+ autoincrement=True,
+ nullable=False,
+ ),
+ sa.Column("issued_at", sa.DateTime(), nullable=True),
+ sa.Column("remote_addr", sa.String(50), nullable=True),
+ sa.Column("user_id", sa.Integer(), nullable=True),
+ # ``action_kind`` carries the high-level avenue that produced
+ # this transaction (``restore`` / ``import`` / ``clone``).
+ # ``NULL`` is the default "ordinary save" — most rows leave
+ # this empty. Commands set
+ # ``session.info["_versioning_action_kind"]`` before commit;
+ # the change-record listener stamps the value here. Parallel
+ # to ``version_changes.entity_kind`` and ``version_changes.kind``
+ # — the schema's third ``*_kind`` column, at transaction scope.
+ sa.Column("action_kind", sa.String(32), nullable=True),
+ )
+
+ if bind.dialect.name == "postgresql":
+ op.execute(
+ "ALTER SEQUENCE version_transaction_id_seq OWNED BY version_transaction.id"
+ )
+
+ # ------------------------------------------------------------------
+ # dashboards_version
+ # ------------------------------------------------------------------
+ op.create_table(
+ "dashboards_version",
+ sa.Column("uuid", UUIDType(binary=True), nullable=True),
+ sa.Column("id", sa.Integer(), nullable=False),
+ sa.Column("dashboard_title", sa.String(500), nullable=True),
+ # ``MediumText()`` mirrors the live column type — on MySQL plain
+ # ``TEXT`` caps at 64 KB, which large dashboards exceed; an
+ # oversized live write would then fail the shadow INSERT under
+ # ``STRICT_TRANS_TABLES`` (or silently truncate without it) and
+ # corrupt the history. Postgres ``TEXT`` is unbounded and SQLite
+ # ignores the length annotation so this is MySQL-driven.
+ sa.Column("position_json", MediumText(), nullable=True),
+ sa.Column("description", sa.Text(), nullable=True),
+ sa.Column("css", MediumText(), nullable=True),
+ sa.Column("theme_id", sa.Integer(), nullable=True),
+ sa.Column("certified_by", sa.Text(), nullable=True),
+ sa.Column("certification_details", sa.Text(), nullable=True),
+ sa.Column("json_metadata", MediumText(), nullable=True),
+ sa.Column("slug", sa.String(255), nullable=True),
+ sa.Column("published", sa.Boolean(), nullable=True),
+ sa.Column("is_managed_externally", sa.Boolean(), nullable=True),
+ sa.Column("external_url", sa.Text(), nullable=True),
+ sa.Column("transaction_id", sa.BigInteger(), nullable=False),
+ sa.Column("end_transaction_id", sa.BigInteger(), nullable=True),
+ sa.Column("operation_type", sa.SmallInteger(), nullable=False),
+ sa.PrimaryKeyConstraint("id", "transaction_id"),
+ sa.ForeignKeyConstraint(
+ ["transaction_id"],
+ ["version_transaction.id"],
+ name="fk_dashboards_version_transaction_id",
+ ),
+ sa.ForeignKeyConstraint(
+ ["end_transaction_id"],
+ ["version_transaction.id"],
+ name="fk_dashboards_version_end_transaction_id",
+ ),
+ )
+ op.create_index(
Review Comment:
Addressed by subsequent commits; the flagged code is no longer current after the rebase onto master. Resolving as outdated.
##########
superset/migrations/versions/2026-05-28_19-50_56cd24c07170_add_versioning_tables.py:
##########
@@ -0,0 +1,562 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""add_versioning_tables
+
+Creates the full schema backing entity versioning in a single
+migration:
+
+1. ``version_transaction`` — audit log keyed by Continuum's per-flush
+ transaction id (plus a Postgres-specific id sequence).
+2. **Parent shadow tables** mirroring each versioned entity's columns:
+ ``dashboards_version`` / ``slices_version`` / ``tables_version``.
+3. ``version_changes`` — field-level diff log keyed to a
+ ``(transaction, entity)`` pair; each row describes one atomic change
+ (one field or one child-collection element) that occurred during a
+ save.
+4. **Child shadow tables** for the collections Continuum auto-registers
+ when ``__versioned__`` is applied to ``TableColumn`` / ``SqlMetric``
+ and the ``slices`` exclude is removed from
+ ``Dashboard.__versioned__``: ``table_columns_version`` /
+ ``sql_metrics_version`` / ``dashboard_slices_version``.
+
+All shadow tables follow the validity-strategy shape (mirrored columns
++ ``transaction_id`` / ``end_transaction_id`` / ``operation_type``
+bookkeeping with FKs to ``version_transaction.id``). The current
+version row has ``end_transaction_id = NULL``.
+
+This migration replaces three iterative migrations from the spike phase
+(``56cd24c07170``, ``e1f3c5a7b9d0``, ``f7a2b3c4d5e6``) that captured the
+same schema in three steps as the feature was developed. Compacting
+gives downstream operators one migration to apply / reverse and one
+review surface. The ``revision`` hash is reused from the original first
+migration so anyone still tracking the chain by that hash lands on the
+same logical change set.
+
+Generated by hand because the current Continuum + Alembic-autogenerate
+interaction trips on the renamed ``transaction`` -> ``version_transaction``
+table key (``KeyError`` lookups in ``table_key_to_table``). Column
+inventories were sourced from the live model ``__table__`` definitions
+and ``version_class(...).__table__`` / Continuum association metadata.
+
+Primary key choice. Both ``version_transaction.id`` and
+``version_changes.id`` are ``BigInteger`` autoincrement — a deliberate
+carveout from the project's UUID-PK convention for new models (see
+``CLAUDE.md`` §"UUID Migration"). ``version_transaction`` is keyed
+externally by SQLAlchemy-Continuum via
+``nextval('version_transaction_id_seq')`` on every INSERT; matching
+that contract is required for ``versioning_manager`` to function.
+``version_changes`` follows the same shape because the user-facing
+identity is the ``(transaction_id, entity_kind, entity_id, sequence)``
+composite unique key, not the row id; the API surfaces a deterministic
+UUIDv5 ``version_uuid`` derived from ``entity.uuid`` and
+``transaction_id`` for stable external references.
+
+Revision ID: 56cd24c07170
+Revises: 2bee73611e32
+Create Date: 2026-05-28 19:50:00.000000
+
+"""
+
+from __future__ import annotations
+
+import sqlalchemy as sa
+from alembic import op
+from sqlalchemy_utils import UUIDType
+
+from superset.utils.core import MediumText
+
+revision = "56cd24c07170"
+# Stacked on the composite-PK association-tables change (2bee73611e32) so the
+# Continuum shadow tables this migration creates can mirror the
+# composite-PK shape of the live association tables. If that change
+# is removed from the stack, this should be reverted to "ce6bd21901ab".
+down_revision = "2bee73611e32"
+
+
+def upgrade() -> None:
+ bind = op.get_bind()
+
+ # ------------------------------------------------------------------
+ # version_transaction
+ #
+ # Audit log for each versioning event. Continuum emits
+ # ``nextval('version_transaction_id_seq')`` on every INSERT, so the
+ # sequence must exist before the table on Postgres. SQLite/MySQL
+ # ignore the explicit CREATE SEQUENCE (they auto-increment natively).
+ # ------------------------------------------------------------------
+ if bind.dialect.name == "postgresql":
+ op.execute("CREATE SEQUENCE IF NOT EXISTS version_transaction_id_seq")
+
+ op.create_table(
+ "version_transaction",
+ sa.Column(
+ "id",
+ sa.BigInteger(),
+ sa.Sequence("version_transaction_id_seq"),
+ primary_key=True,
+ autoincrement=True,
+ nullable=False,
+ ),
+ sa.Column("issued_at", sa.DateTime(), nullable=True),
+ sa.Column("remote_addr", sa.String(50), nullable=True),
+ sa.Column("user_id", sa.Integer(), nullable=True),
+ # ``action_kind`` carries the high-level avenue that produced
+ # this transaction (``restore`` / ``import`` / ``clone``).
+ # ``NULL`` is the default "ordinary save" — most rows leave
+ # this empty. Commands set
+ # ``session.info["_versioning_action_kind"]`` before commit;
+ # the change-record listener stamps the value here. Parallel
+ # to ``version_changes.entity_kind`` and ``version_changes.kind``
+ # — the schema's third ``*_kind`` column, at transaction scope.
+ sa.Column("action_kind", sa.String(32), nullable=True),
+ )
+
+ if bind.dialect.name == "postgresql":
+ op.execute(
+ "ALTER SEQUENCE version_transaction_id_seq OWNED BY version_transaction.id"
+ )
+
+ # ------------------------------------------------------------------
+ # dashboards_version
+ # ------------------------------------------------------------------
+ op.create_table(
+ "dashboards_version",
+ sa.Column("uuid", UUIDType(binary=True), nullable=True),
+ sa.Column("id", sa.Integer(), nullable=False),
+ sa.Column("dashboard_title", sa.String(500), nullable=True),
+ # ``MediumText()`` mirrors the live column type — on MySQL plain
+ # ``TEXT`` caps at 64 KB, which large dashboards exceed; an
+ # oversized live write would then fail the shadow INSERT under
+ # ``STRICT_TRANS_TABLES`` (or silently truncate without it) and
+ # corrupt the history. Postgres ``TEXT`` is unbounded and SQLite
+ # ignores the length annotation so this is MySQL-driven.
+ sa.Column("position_json", MediumText(), nullable=True),
+ sa.Column("description", sa.Text(), nullable=True),
+ sa.Column("css", MediumText(), nullable=True),
+ sa.Column("theme_id", sa.Integer(), nullable=True),
+ sa.Column("certified_by", sa.Text(), nullable=True),
+ sa.Column("certification_details", sa.Text(), nullable=True),
+ sa.Column("json_metadata", MediumText(), nullable=True),
+ sa.Column("slug", sa.String(255), nullable=True),
+ sa.Column("published", sa.Boolean(), nullable=True),
+ sa.Column("is_managed_externally", sa.Boolean(), nullable=True),
+ sa.Column("external_url", sa.Text(), nullable=True),
+ sa.Column("transaction_id", sa.BigInteger(), nullable=False),
+ sa.Column("end_transaction_id", sa.BigInteger(), nullable=True),
+ sa.Column("operation_type", sa.SmallInteger(), nullable=False),
+ sa.PrimaryKeyConstraint("id", "transaction_id"),
+ sa.ForeignKeyConstraint(
+ ["transaction_id"],
+ ["version_transaction.id"],
+ name="fk_dashboards_version_transaction_id",
+ ),
+ sa.ForeignKeyConstraint(
+ ["end_transaction_id"],
+ ["version_transaction.id"],
+ name="fk_dashboards_version_end_transaction_id",
+ ),
+ )
+ op.create_index(
+ "ix_dashboards_version_end_transaction_id",
+ "dashboards_version",
+ ["end_transaction_id"],
+ )
+ op.create_index(
+ "ix_dashboards_version_operation_type",
+ "dashboards_version",
+ ["operation_type"],
+ )
+ op.create_index(
+ "ix_dashboards_version_transaction_id",
+ "dashboards_version",
+ ["transaction_id"],
+ )
+
+ # ------------------------------------------------------------------
+ # slices_version (Charts)
+ # ------------------------------------------------------------------
+ op.create_table(
+ "slices_version",
+ sa.Column("uuid", UUIDType(binary=True), nullable=True),
+ sa.Column("id", sa.Integer(), nullable=False),
+ sa.Column("slice_name", sa.String(250), nullable=True),
+ sa.Column("datasource_id", sa.Integer(), nullable=True),
+ sa.Column("datasource_type", sa.String(200), nullable=True),
+ sa.Column("datasource_name", sa.String(2000), nullable=True),
+ sa.Column("viz_type", sa.String(250), nullable=True),
+ sa.Column("params", MediumText(), nullable=True),
+ sa.Column("description", sa.Text(), nullable=True),
+ sa.Column("cache_timeout", sa.Integer(), nullable=True),
+ sa.Column("certified_by", sa.Text(), nullable=True),
+ sa.Column("certification_details", sa.Text(), nullable=True),
+ sa.Column("is_managed_externally", sa.Boolean(), nullable=True),
+ sa.Column("external_url", sa.Text(), nullable=True),
+ sa.Column("transaction_id", sa.BigInteger(), nullable=False),
+ sa.Column("end_transaction_id", sa.BigInteger(), nullable=True),
+ sa.Column("operation_type", sa.SmallInteger(), nullable=False),
+ sa.PrimaryKeyConstraint("id", "transaction_id"),
+ sa.ForeignKeyConstraint(
+ ["transaction_id"],
+ ["version_transaction.id"],
+ name="fk_slices_version_transaction_id",
+ ),
+ sa.ForeignKeyConstraint(
+ ["end_transaction_id"],
+ ["version_transaction.id"],
+ name="fk_slices_version_end_transaction_id",
+ ),
+ )
+ op.create_index(
+ "ix_slices_version_end_transaction_id",
+ "slices_version",
+ ["end_transaction_id"],
+ )
+ op.create_index(
+ "ix_slices_version_operation_type",
+ "slices_version",
+ ["operation_type"],
+ )
+ op.create_index(
+ "ix_slices_version_transaction_id",
+ "slices_version",
+ ["transaction_id"],
+ )
+
+ # ------------------------------------------------------------------
+ # tables_version (SqlaTable / Datasets)
+ # ------------------------------------------------------------------
+ op.create_table(
+ "tables_version",
+ sa.Column("uuid", UUIDType(binary=True), nullable=True),
+ sa.Column("id", sa.Integer(), nullable=False),
+ sa.Column("description", sa.Text(), nullable=True),
+ sa.Column("default_endpoint", sa.Text(), nullable=True),
+ sa.Column("is_featured", sa.Boolean(), nullable=True),
+ sa.Column("filter_select_enabled", sa.Boolean(), nullable=True),
+ sa.Column("offset", sa.Integer(), nullable=True),
+ sa.Column("cache_timeout", sa.Integer(), nullable=True),
+ sa.Column("params", sa.String(1000), nullable=True),
+ sa.Column("is_managed_externally", sa.Boolean(), nullable=True),
+ sa.Column("external_url", sa.Text(), nullable=True),
+ sa.Column("table_name", sa.String(250), nullable=True),
+ sa.Column("main_dttm_col", sa.String(250), nullable=True),
+ sa.Column("currency_code_column", sa.String(250), nullable=True),
+ sa.Column("database_id", sa.Integer(), nullable=True),
+ sa.Column("fetch_values_predicate", sa.Text(), nullable=True),
+ sa.Column("schema", sa.String(255), nullable=True),
+ sa.Column("catalog", sa.String(256), nullable=True),
+ sa.Column("sql", MediumText(), nullable=True),
+ sa.Column("is_sqllab_view", sa.Boolean(), nullable=True),
+ sa.Column("template_params", sa.Text(), nullable=True),
+ sa.Column("extra", sa.Text(), nullable=True),
+ sa.Column("normalize_columns", sa.Boolean(), nullable=True),
+ sa.Column("always_filter_main_dttm", sa.Boolean(), nullable=True),
+ sa.Column("folders", sa.JSON(), nullable=True),
+ sa.Column("transaction_id", sa.BigInteger(), nullable=False),
+ sa.Column("end_transaction_id", sa.BigInteger(), nullable=True),
+ sa.Column("operation_type", sa.SmallInteger(), nullable=False),
+ sa.PrimaryKeyConstraint("id", "transaction_id"),
+ sa.ForeignKeyConstraint(
+ ["transaction_id"],
+ ["version_transaction.id"],
+ name="fk_tables_version_transaction_id",
+ ),
+ sa.ForeignKeyConstraint(
+ ["end_transaction_id"],
+ ["version_transaction.id"],
+ name="fk_tables_version_end_transaction_id",
+ ),
+ )
+ op.create_index(
+ "ix_tables_version_end_transaction_id",
+ "tables_version",
+ ["end_transaction_id"],
+ )
+ op.create_index(
+ "ix_tables_version_operation_type",
+ "tables_version",
+ ["operation_type"],
+ )
+ op.create_index(
+ "ix_tables_version_transaction_id",
+ "tables_version",
+ ["transaction_id"],
+ )
+
+ # ------------------------------------------------------------------
+ # version_changes
+ #
+ # Field-level diff log keyed to a (transaction, entity) pair. Each
+ # row describes one atomic change (one field or one child-collection
+ # element) that occurred to one entity during a save.
+ #
+ # ``(entity_kind, entity_id)`` is a polymorphic reference: depending
+ # on ``entity_kind`` (``"chart"`` / ``"dashboard"`` / ``"dataset"``)
+ # the ``entity_id`` is the integer PK on ``slices`` / ``dashboards`` /
+ # ``tables`` respectively. SQL has no native polymorphic FK, so the
+ # constraint is intentionally omitted — cleanup relies on the
+ # ``CASCADE`` from ``version_transaction.id`` plus command-layer
+ # ordering for entity deletes (the command that hard-deletes the
+ # entity runs inside the same transaction that prunes its history).
+ # A bare ``DELETE FROM <entity_table> WHERE id = X`` outside that
+ # transactional boundary leaves orphan ``version_changes`` rows
+ # whose ``entity_id`` references a vanished row — the read-side
+ # tombstone-state lookup handles this gracefully.
+ # ------------------------------------------------------------------
+ op.create_table(
+ "version_changes",
+ sa.Column(
+ "id",
+ sa.BigInteger(),
+ primary_key=True,
+ autoincrement=True,
+ nullable=False,
+ ),
+ sa.Column(
+ "transaction_id",
+ sa.BigInteger(),
+ sa.ForeignKey("version_transaction.id", ondelete="CASCADE"),
+ nullable=False,
+ ),
+ sa.Column(
+ "entity_kind",
+ sa.String(length=32),
+ nullable=False,
+ ),
+ sa.Column(
+ "entity_id",
+ sa.Integer(),
+ nullable=False,
+ ),
+ sa.Column(
+ # Integer, not SmallInteger: per-entity sequence within one
+ # transaction is assigned by unbounded enumerate(); a
+ # pathological diff (e.g. a giant position_json rewrite) could
+ # overflow SmallInteger's 32767 on Postgres/MySQL.
+ "sequence",
+ sa.Integer(),
+ nullable=False,
+ ),
+ sa.Column(
+ "kind",
+ sa.String(length=32),
+ nullable=False,
+ ),
+ # ``operation`` is the per-record verb: ``add`` / ``remove`` /
+ # ``move`` / ``edit``. ``move`` only fires for layout records;
+ # the other three apply across every emit site. Made explicit
+ # so consumers don't have to infer the verb from ``from_value``
+ # / ``to_value`` null-tests or from ``path[0]`` for layout records.
+ sa.Column(
+ "operation",
+ sa.String(length=16),
+ nullable=False,
+ ),
+ sa.Column("path", sa.JSON(), nullable=False),
+ sa.Column("from_value", sa.JSON(), nullable=True),
+ sa.Column("to_value", sa.JSON(), nullable=True),
+ sa.UniqueConstraint(
+ "transaction_id",
+ "entity_kind",
+ "entity_id",
+ "sequence",
+ name="uq_version_changes_tx_entity_sequence",
+ ),
+ )
+ op.create_index(
+ "ix_version_changes_kind",
+ "version_changes",
+ ["kind"],
+ )
+ op.create_index(
+ "ix_version_changes_entity",
+ "version_changes",
+ ["entity_kind", "entity_id"],
+ )
+
+ # ------------------------------------------------------------------
+ # table_columns_version
+ # ------------------------------------------------------------------
+ op.create_table(
+ "table_columns_version",
+ sa.Column("uuid", UUIDType(binary=True), nullable=True),
+ sa.Column("id", sa.Integer(), nullable=False),
+ sa.Column("column_name", sa.String(255), nullable=True),
+ sa.Column("verbose_name", sa.String(1024), nullable=True),
+ sa.Column("is_active", sa.Boolean(), nullable=True),
+ sa.Column("type", sa.Text(), nullable=True),
+ sa.Column("advanced_data_type", sa.String(255), nullable=True),
+ sa.Column("groupby", sa.Boolean(), nullable=True),
+ sa.Column("filterable", sa.Boolean(), nullable=True),
+ sa.Column("description", MediumText(), nullable=True),
+ sa.Column("table_id", sa.Integer(), nullable=True),
+ sa.Column("is_dttm", sa.Boolean(), nullable=True),
+ sa.Column("expression", MediumText(), nullable=True),
+ sa.Column("python_date_format", sa.String(255), nullable=True),
+ sa.Column("datetime_format", sa.String(100), nullable=True),
+ sa.Column("extra", sa.Text(), nullable=True),
+ sa.Column("transaction_id", sa.BigInteger(), nullable=False),
+ sa.Column("end_transaction_id", sa.BigInteger(), nullable=True),
+ sa.Column("operation_type", sa.SmallInteger(), nullable=False),
+ sa.PrimaryKeyConstraint("id", "transaction_id"),
+ sa.ForeignKeyConstraint(
+ ["transaction_id"],
+ ["version_transaction.id"],
+ name="fk_table_columns_version_transaction_id",
+ ),
+ sa.ForeignKeyConstraint(
+ ["end_transaction_id"],
+ ["version_transaction.id"],
+ name="fk_table_columns_version_end_transaction_id",
+ ),
+ )
+ op.create_index(
+ "ix_table_columns_version_end_transaction_id",
+ "table_columns_version",
+ ["end_transaction_id"],
+ )
+ op.create_index(
+ "ix_table_columns_version_operation_type",
+ "table_columns_version",
+ ["operation_type"],
+ )
+ op.create_index(
+ "ix_table_columns_version_transaction_id",
+ "table_columns_version",
+ ["transaction_id"],
+ )
+
+ # ------------------------------------------------------------------
+ # sql_metrics_version
+ # ------------------------------------------------------------------
+ op.create_table(
+ "sql_metrics_version",
+ sa.Column("uuid", UUIDType(binary=True), nullable=True),
+ sa.Column("id", sa.Integer(), nullable=False),
+ sa.Column("metric_name", sa.String(255), nullable=True),
+ sa.Column("verbose_name", sa.String(1024), nullable=True),
+ sa.Column("metric_type", sa.String(32), nullable=True),
+ sa.Column("description", MediumText(), nullable=True),
+ sa.Column("d3format", sa.String(128), nullable=True),
+ sa.Column("currency", sa.JSON(), nullable=True),
+ sa.Column("warning_text", sa.Text(), nullable=True),
+ sa.Column("table_id", sa.Integer(), nullable=True),
+ sa.Column("expression", MediumText(), nullable=True),
+ sa.Column("extra", sa.Text(), nullable=True),
+ sa.Column("transaction_id", sa.BigInteger(), nullable=False),
+ sa.Column("end_transaction_id", sa.BigInteger(), nullable=True),
+ sa.Column("operation_type", sa.SmallInteger(), nullable=False),
+ sa.PrimaryKeyConstraint("id", "transaction_id"),
+ sa.ForeignKeyConstraint(
+ ["transaction_id"],
+ ["version_transaction.id"],
+ name="fk_sql_metrics_version_transaction_id",
+ ),
+ sa.ForeignKeyConstraint(
+ ["end_transaction_id"],
+ ["version_transaction.id"],
+ name="fk_sql_metrics_version_end_transaction_id",
+ ),
+ )
+ op.create_index(
+ "ix_sql_metrics_version_end_transaction_id",
+ "sql_metrics_version",
+ ["end_transaction_id"],
+ )
+ op.create_index(
+ "ix_sql_metrics_version_operation_type",
+ "sql_metrics_version",
+ ["operation_type"],
+ )
+ op.create_index(
+ "ix_sql_metrics_version_transaction_id",
+ "sql_metrics_version",
+ ["transaction_id"],
+ )
+
+ # ------------------------------------------------------------------
+ # dashboard_slices_version (M2M association)
+ #
+ # The live ``dashboard_slices`` table is reshaped to a
+ # composite PK on ``(dashboard_id, slice_id)`` — no surrogate ``id``.
+ # Continuum auto-mirrors the live columns into the shadow Table at
+ # ``make_versioned()`` time, so the shadow's SQLAlchemy metadata
+ # also has no ``id``. The DB shadow PK is the natural composite key
+ # plus Continuum's bookkeeping (``transaction_id``, ``operation_type``);
+ # ``operation_type`` is included because a single transaction can in
+ # principle produce both INSERT and DELETE shadows for the same
+ # ``(dashboard_id, slice_id)`` pair (slice removed and re-added in
+ # one save).
+ #
+ # If that reshape is removed from the stack, the live table reverts to
+ # carrying its surrogate ``id`` and this migration would need to
+ # match.
+ # ------------------------------------------------------------------
+ op.create_table(
+ "dashboard_slices_version",
+ sa.Column("dashboard_id", sa.Integer(), nullable=False),
+ sa.Column("slice_id", sa.Integer(), nullable=False),
+ sa.Column("transaction_id", sa.BigInteger(), nullable=False),
+ sa.Column("end_transaction_id", sa.BigInteger(), nullable=True),
+ sa.Column("operation_type", sa.SmallInteger(), nullable=False),
+ sa.PrimaryKeyConstraint(
+ "dashboard_id", "slice_id", "transaction_id", "operation_type"
+ ),
+ sa.ForeignKeyConstraint(
+ ["transaction_id"],
+ ["version_transaction.id"],
+ name="fk_dashboard_slices_version_transaction_id",
+ ),
+ sa.ForeignKeyConstraint(
+ ["end_transaction_id"],
+ ["version_transaction.id"],
+ name="fk_dashboard_slices_version_end_transaction_id",
+ ),
+ )
+ op.create_index(
+ "ix_dashboard_slices_version_end_transaction_id",
+ "dashboard_slices_version",
+ ["end_transaction_id"],
+ )
+ op.create_index(
+ "ix_dashboard_slices_version_operation_type",
+ "dashboard_slices_version",
+ ["operation_type"],
+ )
+ op.create_index(
+ "ix_dashboard_slices_version_transaction_id",
+ "dashboard_slices_version",
+ ["transaction_id"],
+ )
+
+
+def downgrade() -> None:
+ # Drop in reverse dependency order: children with FKs to
+ # ``version_transaction`` drop first; ``version_transaction`` and its
+ # sequence drop last.
+ op.drop_table("dashboard_slices_version")
Review Comment:
Addressed by subsequent commits; the flagged code is no longer current after the rebase onto master. Resolving as outdated.
##########
superset/initialization/__init__.py:
##########
@@ -619,6 +619,181 @@ def init_extensions(self) -> None:
extension.manifest.id,
)
+ @staticmethod
+ def _remove_continuum_write_listeners() -> None:
+ """Detach SQLAlchemy-Continuum's own write listeners.
+
+ ``make_versioned()`` runs unconditionally at import of
+ ``superset.extensions`` and registers Continuum's mapper, session,
+ and engine listeners — the ones that write shadow rows and
+ ``version_transaction`` rows on every flush. Skipping only the
+ custom baseline/change-record listeners would leave those running,
+ so with the kill-switch off the shadow tables would silently keep
+ accumulating, contradicting the documented contract.
+
+ This is deliberately a *targeted subset* of
+ ``sqlalchemy_continuum.remove_versioning()``: that helper also
+ calls ``manager.reset()``, which clears ``version_class_map`` —
+ and ``version_class()`` would then silently return the live model
+ class, breaking the read-only ``/versions/`` endpoints this flag
+ promises to keep working.
+
+ Idempotent: guarded on a representative listener so repeated app
+ initializations in one process (test fixtures) don't raise on
+ double-removal.
+ """
+ # pylint: disable=import-outside-toplevel
+ import sqlalchemy as sa
+ from sqlalchemy_continuum import versioning_manager
+
+ if not sa.event.contains(
+ sa.orm.Mapper, "after_insert", versioning_manager.track_inserts
+ ):
+ return # already detached by a prior init
+ versioning_manager.remove_operations_tracking(sa.orm.Mapper)
+ versioning_manager.remove_session_tracking(sa.orm.session.Session)
+ sa.event.remove(
+ sa.engine.Engine,
+ "before_execute",
+ versioning_manager.track_association_operations,
+ )
+ sa.event.remove(
+ sa.engine.Engine, "rollback", versioning_manager.clear_connection
+ )
+ sa.event.remove(
+ sa.engine.Engine,
+ "set_connection_execution_options",
+ versioning_manager.track_cloned_connections,
+ )
+
+ # Belt-and-suspenders: flip Continuum's master option off as well.
+ # Every write listener checks ``manager.options['versioning']`` before
+ # doing work (manager.py / unit_of_work.py), so if a future Continuum
+ # version registers an additional write listener this detach does not
+ # know to remove, that listener still no-ops. ``version_class()`` reads
+ # from ``version_class_map`` and ignores this option, so the read-only
+ # ``/versions/`` endpoints are unaffected.
+ versioning_manager.options["versioning"] = False
+
+ # Verify the known write listeners are actually gone. A Continuum
+ # upgrade that renamed a handler would make the removals above silently
+ # miss, leaving capture half-on while we report "disabled"; surface
+ # that rather than booting in a contradictory state.
+ if sa.event.contains(
+ sa.orm.Mapper, "after_insert", versioning_manager.track_inserts
+ ):
+ logger.warning(
+ "versioning: Continuum write listeners still attached after "
+ "detach; capture may not be fully disabled. This usually means "
+ "the pinned sqlalchemy-continuum version changed how it "
+ "registers listeners."
+ )
+
+ def init_versioning(self) -> None:
+ """Register SQLAlchemy-Continuum baseline and retention listeners.
Review Comment:
Addressed by subsequent commits; the flagged code is no longer current after the rebase onto master. Resolving as outdated.
##########
superset/initialization/__init__.py:
##########
@@ -619,6 +619,181 @@ def init_extensions(self) -> None:
extension.manifest.id,
)
+ @staticmethod
+ def _remove_continuum_write_listeners() -> None:
+ """Detach SQLAlchemy-Continuum's own write listeners.
+
+ ``make_versioned()`` runs unconditionally at import of
+ ``superset.extensions`` and registers Continuum's mapper, session,
+ and engine listeners — the ones that write shadow rows and
+ ``version_transaction`` rows on every flush. Skipping only the
+ custom baseline/change-record listeners would leave those running,
+ so with the kill-switch off the shadow tables would silently keep
+ accumulating, contradicting the documented contract.
+
+ This is deliberately a *targeted subset* of
+ ``sqlalchemy_continuum.remove_versioning()``: that helper also
+ calls ``manager.reset()``, which clears ``version_class_map`` —
+ and ``version_class()`` would then silently return the live model
+ class, breaking the read-only ``/versions/`` endpoints this flag
+ promises to keep working.
+
+ Idempotent: guarded on a representative listener so repeated app
+ initializations in one process (test fixtures) don't raise on
+ double-removal.
+ """
+ # pylint: disable=import-outside-toplevel
+ import sqlalchemy as sa
+ from sqlalchemy_continuum import versioning_manager
+
+ if not sa.event.contains(
+ sa.orm.Mapper, "after_insert", versioning_manager.track_inserts
+ ):
+ return # already detached by a prior init
+ versioning_manager.remove_operations_tracking(sa.orm.Mapper)
+ versioning_manager.remove_session_tracking(sa.orm.session.Session)
+ sa.event.remove(
+ sa.engine.Engine,
+ "before_execute",
+ versioning_manager.track_association_operations,
+ )
+ sa.event.remove(
+ sa.engine.Engine, "rollback", versioning_manager.clear_connection
+ )
+ sa.event.remove(
+ sa.engine.Engine,
+ "set_connection_execution_options",
+ versioning_manager.track_cloned_connections,
+ )
+
+ # Belt-and-suspenders: flip Continuum's master option off as well.
+ # Every write listener checks ``manager.options['versioning']`` before
+ # doing work (manager.py / unit_of_work.py), so if a future Continuum
+ # version registers an additional write listener this detach does not
+ # know to remove, that listener still no-ops. ``version_class()`` reads
+ # from ``version_class_map`` and ignores this option, so the read-only
+ # ``/versions/`` endpoints are unaffected.
+ versioning_manager.options["versioning"] = False
+
+ # Verify the known write listeners are actually gone. A Continuum
+ # upgrade that renamed a handler would make the removals above silently
+ # miss, leaving capture half-on while we report "disabled"; surface
+ # that rather than booting in a contradictory state.
+ if sa.event.contains(
+ sa.orm.Mapper, "after_insert", versioning_manager.track_inserts
+ ):
+ logger.warning(
+ "versioning: Continuum write listeners still attached after "
+ "detach; capture may not be fully disabled. This usually means "
+ "the pinned sqlalchemy-continuum version changed how it "
+ "registers listeners."
+ )
+
+ def init_versioning(self) -> None:
+ """Register SQLAlchemy-Continuum baseline and retention listeners.
+
+ Must be called after all versioned model classes have been imported so
+ that VERSIONED_MODELS can be populated and configure_mappers() has run.
+
+ ``ENABLE_VERSIONING_CAPTURE`` (ships default ``False``) gates the two
+ before-flush listener registrations. The flag is operational, not
+ feature: with it off the infrastructure is inert (no save writes
+ shadow rows); flipping it on activates capture. The switch also lets
+ an operator who observes a versioning-induced regression (e.g. a
+ save-path slowdown attributable to the change-record listener)
+ disable capture in ``superset_config.py`` and restart workers — a
+ 30-second recovery instead of revert-and-redeploy. Shadow tables
+ already created by the migration stay; they just stop accumulating
+ new rows.
+
+ The fallback here is ``False`` so that any app-factory path that
+ does not load ``superset.config`` (some test factories, embedded
+ use) stays inert by default rather than silently enabling capture.
+ """
+ if not self.config.get("ENABLE_VERSIONING_CAPTURE", False):
+ logger.warning(
+ "versioning: ENABLE_VERSIONING_CAPTURE is False; "
+ "skipping baseline + change-record listener registration "
+ "and detaching Continuum's write listeners. Save-path "
+ "capture is disabled; existing shadow tables and "
+ "/versions/ endpoints continue to work read-only."
+ )
+ self._remove_continuum_write_listeners()
+ return
+
+ # Symmetric with the OFF branch's ``options['versioning'] = False``:
+ # re-assert it on here so capture is restored even if a prior app
+ # init in the same process (multi-app / test reentrancy) flipped the
+ # process-global Continuum option off. Without this, an OFF app
+ # initialized before an ON app would leave the option False and the
+ # baseline listener — which gates on it — would silently write no
+ # baselines despite capture being "enabled".
+ from sqlalchemy_continuum import versioning_manager
+
+ versioning_manager.options["versioning"] = True
Review Comment:
Addressed by subsequent commits; the flagged code is no longer current after the rebase onto master. Resolving as outdated.
##########
superset/datasets/api.py:
##########
@@ -440,17 +504,69 @@ def put(self, pk: int) -> Response:
# This validates custom Schema with custom validations
except ValidationError as error:
return self.response_400(message=error.messages)
+
+ # Live version identifiers before the update (empty + query-free when
+ # ``ENABLE_VERSIONING_CAPTURE`` is off).
+ old_info = current_entity_version_info(SqlaTable, pk)
+
try:
+ # Two commands, two commits, two Continuum transactions for an
+ # ``override_columns`` save — deliberately NOT merged into one
+ # transaction. A single-transaction design was attempted and
+ # reverted: ``DBEventLogger`` writes request logs through the
+ # SHARED scoped session and calls ``commit()`` /
+ # ``rollback()`` on it mid-request (superset/utils/log.py),
+ # so any save held uncommitted across a logged sub-action can
+ # be committed half-done (Postgres/MySQL) or rolled back
+ # entirely on a transient logger failure (SQLite's
+ # "database is locked"). Until the event logger gets its own
+ # session, per-command commit boundaries are the only shape
+ # whose failure modes are honest. Consequence the
+ # version-history UI must tolerate: one logical save can
+ # surface as two version transactions stamped the same second.
changed_model = UpdateDatasetCommand(pk, item, override_columns).run()
+ # Capture the post-update identifiers BEFORE the refresh:
+ # RefreshDatasetCommand commits its own transaction, so reading
+ # afterwards would attribute the refresh's version to the
+ # user's update (and old→new would span two transactions).
+ new_info = current_entity_version_info(
+ SqlaTable, changed_model.id, changed_model.uuid
+ )
+ etag_version_uuid = new_info.version_uuid
if override_columns:
RefreshDatasetCommand(pk).run()
- response = self.response(200, id=changed_model.id, result=item)
+ # The ETag must reflect the entity's *current live* version,
+ # which after the refresh is the refresh's transaction —
+ # re-read it rather than reusing the pre-refresh uuid.
+ etag_version_uuid = current_entity_etag_uuid(
+ SqlaTable, changed_model.id, changed_model.uuid
+ )
+ response = self.response(
+ 200,
+ id=changed_model.id,
+ result=item,
+ old_version=old_info.version,
+ new_version=new_info.version,
+ old_transaction_id=old_info.transaction_id,
+ new_transaction_id=new_info.transaction_id,
+ old_version_uuid=old_info.version_uuid,
+ new_version_uuid=new_info.version_uuid,
+ )
+ set_version_etag(response, etag_version_uuid)
except DatasetNotFoundError:
response = self.response_404()
Review Comment:
Addressed by subsequent commits; the flagged code is no longer current after the rebase onto master. Resolving as outdated.
##########
scripts/seed_junction_load.py:
##########
@@ -0,0 +1,679 @@
+#!/usr/bin/env python3
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+# ----------------------------------------------------------------------
+# Stress-test data generator for the composite-PK migration (sc-105349).
+#
+# Bulk-inserts synthetic parent rows and many-to-many junction rows for
+# the eight association tables that the composite-PK migration touches.
Review Comment:
Addressed by subsequent commits; the flagged code is no longer current after the rebase onto master. Resolving as outdated.
##########
scripts/seed_junction_load.py:
##########
@@ -0,0 +1,679 @@
+#!/usr/bin/env python3
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+# ----------------------------------------------------------------------
+# Stress-test data generator for the composite-PK migration (sc-105349).
+#
+# Bulk-inserts synthetic parent rows and many-to-many junction rows for
+# the eight association tables that the composite-PK migration touches.
+# Useful for measuring migration runtime at varying scales — run this at
+# 100K / 1M / 5M / 10M rows and time the migration at each scale to
+# verify the O(N log N) extrapolation.
+#
+# Idempotent: rerunning with the same target is a no-op; rerunning with
+# a higher target adds rows up to the new total. Batched bulk INSERTs
+# (10K rows per statement) make it fast on Postgres, MySQL, and SQLite.
+#
+# Usage (inside the Superset container):
+#
+# docker exec superset-superset-1 \\
+# /app/.venv/bin/python /app/scripts/seed_junction_load.py \\
+# --dashboard-slices 1000000 \\
+# --slice-user 100000 \\
+# --dashboard-user 100000
+#
+# Run with no flags for the defaults shown below. Use ``--dry-run`` to
+# print the planned inserts without writing anything.
+#
+# The script connects via Superset's standard ``DATABASE_*`` env vars
+# (or ``SUPERSET__SQLALCHEMY_DATABASE_URI`` if set), so it works
+# automatically inside the Superset container regardless of which
+# metadata DB backend is in use.
+
+from __future__ import annotations
+
+import argparse
+import logging
+import os
+import sys
+import time
+from contextlib import contextmanager
+from typing import Iterator
+from uuid import uuid4
+
+import sqlalchemy as sa
+from sqlalchemy.engine import Connection, Engine
+
+logger = logging.getLogger("seed_junction_load")
+
+# Bulk INSERT batch size. Larger values = fewer statements but more memory.
+BATCH = 10_000
+
+# Default per-junction-table target row counts. Tuned to mimic the shape
+# of a large multi-team Superset install. Override via CLI flags.
+DEFAULTS: dict[str, int] = {
+ "dashboard_slices": 1_000_000,
+ "slice_user": 100_000,
+ "dashboard_user": 100_000,
+ "dashboard_roles": 10_000,
+}
+
+# (junction_table, fk1_col, fk2_col, parent1_table, parent2_table)
+# parents reference id columns; we generate (fk1, fk2) pairs by sampling
+# from the parents' existing IDs.
+JUNCTIONS: list[tuple[str, str, str, str, str]] = [
+ ("dashboard_slices", "dashboard_id", "slice_id", "dashboards", "slices"),
+ ("slice_user", "user_id", "slice_id", "ab_user", "slices"),
+ ("dashboard_user", "user_id", "dashboard_id", "ab_user", "dashboards"),
+ ("dashboard_roles", "dashboard_id", "role_id", "dashboards", "ab_role"),
+]
+
+# Junction tables that originally carried ``UNIQUE(fk1, fk2)`` and therefore
+# cannot accept duplicate ``(fk1, fk2)`` pairs even on the pre-migration
+# (downgrade) schema. The other JUNCTIONS allow duplicates pre-migration.
+# Only ``dashboard_slices`` is listed: the migration's other UNIQUE table
+# (``report_schedule_user``) is not in JUNCTIONS — this script doesn't seed
+# it — so listing it here would imply coverage that doesn't exist. Add it
+# alongside a JUNCTIONS entry if that table ever gets seeded.
+JUNCTIONS_WITH_UNIQUE: set[str] = {"dashboard_slices"}
+
+
+# ----------------------------------------------------------------------
+# Connection setup
+# ----------------------------------------------------------------------
+
+
+def build_engine() -> Engine:
+ """Build a SQLAlchemy engine from Superset env vars."""
+ if uri := os.environ.get("SUPERSET__SQLALCHEMY_DATABASE_URI"):
+ logger.info("Using SUPERSET__SQLALCHEMY_DATABASE_URI from env")
+ return sa.create_engine(uri)
+
+ try:
+ dialect = os.environ["DATABASE_DIALECT"]
+ user = os.environ["DATABASE_USER"]
+ password = os.environ["DATABASE_PASSWORD"]
+ host = os.environ["DATABASE_HOST"]
+ port = os.environ["DATABASE_PORT"]
+ db = os.environ["DATABASE_DB"]
+ except KeyError as exc:
+ sys.exit(
+ f"Missing env var {exc}; either set DATABASE_DIALECT/USER/PASSWORD/"
+ f"HOST/PORT/DB or SUPERSET__SQLALCHEMY_DATABASE_URI before running."
+ )
+
+ uri = f"{dialect}://{user}:{password}@{host}:{port}/{db}"
Review Comment:
Addressed by subsequent commits; the flagged code is no longer current after the rebase onto master. Resolving as outdated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(versioning): cross-entity version activity view [superset]
Posted by "mikebridge (via GitHub)" <gi...@apache.org>.
mikebridge commented on code in PR #41076:
URL: https://github.com/apache/superset/pull/41076#discussion_r3574997591
##########
scripts/seed_junction_load.py:
##########
@@ -0,0 +1,679 @@
+#!/usr/bin/env python3
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+# ----------------------------------------------------------------------
+# Stress-test data generator for the composite-PK migration (sc-105349).
+#
+# Bulk-inserts synthetic parent rows and many-to-many junction rows for
+# the eight association tables that the composite-PK migration touches.
+# Useful for measuring migration runtime at varying scales — run this at
+# 100K / 1M / 5M / 10M rows and time the migration at each scale to
+# verify the O(N log N) extrapolation.
+#
+# Idempotent: rerunning with the same target is a no-op; rerunning with
+# a higher target adds rows up to the new total. Batched bulk INSERTs
+# (10K rows per statement) make it fast on Postgres, MySQL, and SQLite.
+#
+# Usage (inside the Superset container):
+#
+# docker exec superset-superset-1 \\
+# /app/.venv/bin/python /app/scripts/seed_junction_load.py \\
+# --dashboard-slices 1000000 \\
+# --slice-user 100000 \\
+# --dashboard-user 100000
+#
+# Run with no flags for the defaults shown below. Use ``--dry-run`` to
+# print the planned inserts without writing anything.
+#
+# The script connects via Superset's standard ``DATABASE_*`` env vars
+# (or ``SUPERSET__SQLALCHEMY_DATABASE_URI`` if set), so it works
+# automatically inside the Superset container regardless of which
+# metadata DB backend is in use.
+
+from __future__ import annotations
+
+import argparse
+import logging
+import os
+import sys
+import time
+from contextlib import contextmanager
+from typing import Iterator
+from uuid import uuid4
+
+import sqlalchemy as sa
+from sqlalchemy.engine import Connection, Engine
+
+logger = logging.getLogger("seed_junction_load")
+
+# Bulk INSERT batch size. Larger values = fewer statements but more memory.
+BATCH = 10_000
+
+# Default per-junction-table target row counts. Tuned to mimic the shape
+# of a large multi-team Superset install. Override via CLI flags.
+DEFAULTS: dict[str, int] = {
+ "dashboard_slices": 1_000_000,
+ "slice_user": 100_000,
+ "dashboard_user": 100_000,
+ "dashboard_roles": 10_000,
+}
+
+# (junction_table, fk1_col, fk2_col, parent1_table, parent2_table)
+# parents reference id columns; we generate (fk1, fk2) pairs by sampling
+# from the parents' existing IDs.
+JUNCTIONS: list[tuple[str, str, str, str, str]] = [
+ ("dashboard_slices", "dashboard_id", "slice_id", "dashboards", "slices"),
+ ("slice_user", "user_id", "slice_id", "ab_user", "slices"),
+ ("dashboard_user", "user_id", "dashboard_id", "ab_user", "dashboards"),
+ ("dashboard_roles", "dashboard_id", "role_id", "dashboards", "ab_role"),
+]
+
+# Junction tables that originally carried ``UNIQUE(fk1, fk2)`` and therefore
+# cannot accept duplicate ``(fk1, fk2)`` pairs even on the pre-migration
+# (downgrade) schema. The other JUNCTIONS allow duplicates pre-migration.
+# Only ``dashboard_slices`` is listed: the migration's other UNIQUE table
+# (``report_schedule_user``) is not in JUNCTIONS — this script doesn't seed
+# it — so listing it here would imply coverage that doesn't exist. Add it
+# alongside a JUNCTIONS entry if that table ever gets seeded.
+JUNCTIONS_WITH_UNIQUE: set[str] = {"dashboard_slices"}
+
+
+# ----------------------------------------------------------------------
+# Connection setup
+# ----------------------------------------------------------------------
+
+
+def build_engine() -> Engine:
+ """Build a SQLAlchemy engine from Superset env vars."""
+ if uri := os.environ.get("SUPERSET__SQLALCHEMY_DATABASE_URI"):
+ logger.info("Using SUPERSET__SQLALCHEMY_DATABASE_URI from env")
+ return sa.create_engine(uri)
+
+ try:
+ dialect = os.environ["DATABASE_DIALECT"]
+ user = os.environ["DATABASE_USER"]
+ password = os.environ["DATABASE_PASSWORD"]
+ host = os.environ["DATABASE_HOST"]
+ port = os.environ["DATABASE_PORT"]
+ db = os.environ["DATABASE_DB"]
+ except KeyError as exc:
+ sys.exit(
+ f"Missing env var {exc}; either set DATABASE_DIALECT/USER/PASSWORD/"
+ f"HOST/PORT/DB or SUPERSET__SQLALCHEMY_DATABASE_URI before running."
+ )
+
+ uri = f"{dialect}://{user}:{password}@{host}:{port}/{db}"
+ logger.info(
+ "Built URI from DATABASE_* env vars (dialect=%s, host=%s)", dialect, host
+ )
+ return sa.create_engine(uri)
+
+
+# ----------------------------------------------------------------------
+# Helpers
+# ----------------------------------------------------------------------
+
+
+def uuid_value(dialect_name: str) -> bytes | str:
+ """Return a UUID in the form the active dialect expects.
+
+ MySQL stores UUIDs as ``BINARY(16)`` (16 raw bytes); Postgres has a
+ native ``UUID`` type that accepts strings; SQLite stores them as
+ BLOB/TEXT and accepts either. Branching here keeps the seed script
+ backend-agnostic without depending on Superset's custom column types.
+ """
+ if dialect_name.startswith("mysql"):
+ return uuid4().bytes
+ return str(uuid4())
+
+
+@contextmanager
+def time_phase(name: str) -> Iterator[None]:
+ """Log elapsed wall time for a named phase."""
+ start = time.monotonic()
+ logger.info("[%s] starting", name)
+ try:
+ yield
+ finally:
+ elapsed = time.monotonic() - start
+ logger.info("[%s] done in %.2fs", name, elapsed)
+
+
+def count_rows(conn: Connection, table: str) -> int:
+ return conn.scalar(sa.text(f"SELECT COUNT(*) FROM {table}")) or 0 # noqa: S608
+
+
+def existing_ids(conn: Connection, table: str, limit: int | None = None) -> list[int]:
+ sql = f"SELECT id FROM {table} ORDER BY id" # noqa: S608
+ if limit is not None:
+ sql += f" LIMIT {limit}"
+ return [row[0] for row in conn.execute(sa.text(sql))]
+
+
+# ----------------------------------------------------------------------
+# Parent seeders
+#
+# Each function ensures the named parent table has at least ``target``
+# rows by inserting synthetic ones with minimal-but-valid columns.
+# Returns nothing; subsequent code reads back IDs via ``existing_ids``.
+# ----------------------------------------------------------------------
+
+
+def seed_dashboards(conn: Connection, target: int, dry_run: bool) -> None:
+ current = count_rows(conn, "dashboards")
+ if current >= target:
+ logger.info(
+ "dashboards: %d rows (target %d) — no insert needed", current, target
Review Comment:
Addressed by subsequent commits; the flagged code is no longer current after the rebase onto master. Resolving as outdated.
##########
superset/versioning/activity/orchestrator.py:
##########
@@ -0,0 +1,434 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Top-level orchestrator + query-param parsing + observability.
+
+This is the public entry point for the activity-view read path. One
+function — :func:`get_activity` — dispatches on the path entity's
+model class to assemble the cross-entity activity stream:
+
+1. ``resolve_path_entity`` (queries.py) — resolve UUID → live entity.
+2. ``resolve_scope`` (scope.py) — build the related-entity window list.
+3. ``fetch_change_records`` (queries.py) — pull rows from
+ ``version_changes`` joined with ``version_transaction`` and ``ab_user``.
+4. ``filter_records_by_visibility`` (visibility.py) — silent
+ drop of records the requester can't read.
+5. ``apply_entity_name_denormalization`` (queries.py) — resolve entity names
+ from the shadow row valid at each record's transaction_id.
+6. ``apply_record_decoration`` (render.py) — synthesize the ActivityRecord
+ DTO fields and strip internal-only columns.
+7. Paginate in Python over the post-filter list.
+
+Parameter parsing for the REST endpoints lives here too —
+:func:`parse_activity_query_params` is called by the three
+``/activity/`` endpoint handlers before they call ``get_activity``.
+Same for the observability instrumentation: ``_phase_timer`` and
+``_emit_request_shape_attributes`` emit the per-phase timing and
+request-shape metrics, on the same prefix the cross-coupling test pins.
+"""
+
+from __future__ import annotations
+
+import contextlib
+from collections.abc import Iterator
+from datetime import datetime, timezone
+from typing import Any
+from uuid import UUID
+
+from flask import Response
+from flask_appbuilder import Model
+
+from superset.utils import json
+from superset.versioning.activity.kinds import EntityWindows
+from superset.versioning.activity.queries import (
+ apply_entity_name_denormalization,
+ fetch_change_records,
+ first_tracked_tx,
+ mark_first_tracked_saves,
+ resolve_path_entity,
+)
+from superset.versioning.activity.render import apply_record_decoration
+from superset.versioning.activity.scope import resolve_scope
+from superset.versioning.activity.visibility import filter_records_by_visibility
+from superset.versioning.api_helpers import (
+ PathEntityResponseError,
+ resolve_endpoint_path_entity,
+)
+
+_DEFAULT_PAGE_SIZE = 25
+_MAX_PAGE_SIZE = 200
+_VALID_INCLUDE_VALUES: frozenset[str] = frozenset({"self", "related", "all"})
+
+
+# Upper bound on the ``q`` search string. The search is a substring scan over
+# the (already-capped) materialized record set, so this is a cheap-DoS guard,
+# not a correctness limit.
+_MAX_Q_LENGTH = 1024
Review Comment:
Addressed by subsequent commits; the flagged code is no longer current after the rebase onto master. Resolving as outdated.
##########
superset/versioning/activity/windows.py:
##########
@@ -0,0 +1,108 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Pure window arithmetic on half-open ``[start_tx, end_tx)`` intervals.
+
+Extracted from the DB-touching scope resolution so that:
+
+* :mod:`scope` (DB-touching) can import this module at module-top.
+* :mod:`queries.fetch_change_records` can import
+ :func:`row_within_any_window` at module-top instead of through a
+ lazy import that previously dodged a ``scope ↔ queries`` cycle.
+
+Everything here is pure Python — no DB, no Flask. ``end_tx = None``
+means "open-ended (current)" and behaves like positive infinity.
+"""
+
+from __future__ import annotations
+
+from typing import Any
+
+from superset.versioning.activity.kinds import EntityWindows, Window
+
+
+def intersect_windows(outer: Window, inner: Window) -> Window | None:
+ """Intersect two half-open ``[start_tx, end_tx)`` windows.
+
+ Returns the clipped overlap, or ``None`` when they are disjoint.
+ ``end_tx = None`` means "open ended (current)" and acts like
+ positive infinity. Thin wrapper over :meth:`Window.intersect` —
+ kept as a free function so callers and tests don't have to migrate
+ to method form in lockstep with the dataclass promotion.
+ """
+ return outer.intersect(inner)
+
+
+def row_within_any_window(row: dict[str, Any], windows: list[Window]) -> bool:
+ """``True`` iff ``row['transaction_id']`` falls inside at least one
+ of *windows*. Half-open interval semantics match
+ :func:`intersect_windows`."""
+ if not windows:
+ return False
+ tx_id = row["transaction_id"]
Review Comment:
Addressed by subsequent commits; the flagged code is no longer current after the rebase onto master. Resolving as outdated.
##########
superset/versioning/activity/windows.py:
##########
@@ -0,0 +1,108 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Pure window arithmetic on half-open ``[start_tx, end_tx)`` intervals.
+
+Extracted from the DB-touching scope resolution so that:
+
+* :mod:`scope` (DB-touching) can import this module at module-top.
+* :mod:`queries.fetch_change_records` can import
+ :func:`row_within_any_window` at module-top instead of through a
+ lazy import that previously dodged a ``scope ↔ queries`` cycle.
+
+Everything here is pure Python — no DB, no Flask. ``end_tx = None``
+means "open-ended (current)" and behaves like positive infinity.
+"""
+
+from __future__ import annotations
+
+from typing import Any
+
+from superset.versioning.activity.kinds import EntityWindows, Window
+
+
+def intersect_windows(outer: Window, inner: Window) -> Window | None:
+ """Intersect two half-open ``[start_tx, end_tx)`` windows.
+
+ Returns the clipped overlap, or ``None`` when they are disjoint.
+ ``end_tx = None`` means "open ended (current)" and acts like
+ positive infinity. Thin wrapper over :meth:`Window.intersect` —
+ kept as a free function so callers and tests don't have to migrate
+ to method form in lockstep with the dataclass promotion.
+ """
+ return outer.intersect(inner)
+
+
+def row_within_any_window(row: dict[str, Any], windows: list[Window]) -> bool:
+ """``True`` iff ``row['transaction_id']`` falls inside at least one
+ of *windows*. Half-open interval semantics match
+ :func:`intersect_windows`."""
+ if not windows:
+ return False
+ tx_id = row["transaction_id"]
+ return any(w.contains(tx_id) for w in windows)
+
+
+def merge_entity_windows(scope: list[EntityWindows]) -> list[EntityWindows]:
+ """Collapse repeated ``(api_kind, entity_id)`` entries by unioning
+ their window lists, and collapse overlapping/touching windows
+ within each entity into one.
+
+ The OR-clause in
+ :func:`~superset.versioning.activity.queries.fetch_change_records`
+ generates one branch per (kind, id, window) tuple. Without the
+ within-entity union, a chart that's been attached-and-detached
+ many times (or that repeated fixture loads have populated the M2M
+ shadow for) yields a separate clause per redundant window — at
+ ~10 entities × ~50 windows the SQL hits SQLite's
+ ``SQLITE_MAX_EXPR_DEPTH`` (1000). Merging here keeps the clause
+ count proportional to the number of *distinct* validity intervals,
+ not the number of shadow rows.
+ """
+ merged: dict[tuple[str, int], list[Window]] = {}
+ for api_kind, entity_id, windows in scope:
+ merged.setdefault((api_kind, entity_id), []).extend(windows)
+ return [
+ (api_kind, entity_id, union_windows(windows))
+ for (api_kind, entity_id), windows in merged.items()
+ ]
+
+
+def union_windows(windows: list[Window]) -> list[Window]:
+ """Sort + merge overlapping/touching half-open intervals.
+
+ Pure function — no DB. Touching ``[a, b)`` and ``[b, c)`` merge into
+ ``[a, c)``. ``end_tx = None`` (open-ended) absorbs everything to its
+ right. Returns a minimal disjoint cover of the input set.
+ """
+ if not windows:
+ return []
+ sorted_windows = sorted(windows, key=lambda w: w.start_tx)
Review Comment:
Addressed by subsequent commits; the flagged code is no longer current after the rebase onto master. Resolving as outdated.
##########
superset/versioning/changes/listener.py:
##########
@@ -198,6 +198,29 @@ def build_action_headline(
# is correctly deduped.
_REGISTERED_SENTINEL = "_versioning_change_listener_registered"
+#: Metric namespace for swallowed capture-path failures. The capture
+#: listeners fail open (a versioning bug must never break a user's save),
+#: so the read path (``activity/orchestrator``) is richly instrumented but
+#: the write path historically logged-and-swallowed with no counter. Each
+#: ``_incr_capture_error(stage)`` emits ``<prefix>.<stage>.error`` so a
+#: systematic capture regression is alertable rather than log-grep-only.
+_CAPTURE_METRIC_PREFIX = "superset.versioning.capture"
Review Comment:
Addressed by subsequent commits; the flagged code is no longer current after the rebase onto master. Resolving as outdated.
##########
tests/integration_tests/versioning/activity_view_tests.py:
##########
@@ -0,0 +1,1250 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Integration tests for the cross-entity activity-view API (sc-107283).
+
+US1 — dashboard activity stream: ``GET /api/v1/dashboard/<uuid>/activity/``.
+Tests for US2 (chart activity) and US3 (dataset activity) come in later
+phases.
+
+Per spec T053 / sc-103156 T062, every test that mutates a fixture entity
+wraps the test body in ``try``/``finally`` with
+``metadata_db.session.rollback()`` in the ``finally``. The rationale is
+documented in the spec — Continuum captures dirty mappers during
+autoflush, so leaving an instrumented attribute dirty pollutes
+downstream tests via the shadow tables.
+"""
+
+from __future__ import annotations
+
+from typing import Any
+from uuid import uuid4
+
+import pytest
+
+from superset.connectors.sqla.models import SqlaTable
+from superset.extensions import db
+from superset.models.dashboard import Dashboard
+from superset.models.slice import Slice
+from superset.utils import json as _json
+from tests.integration_tests.base_tests import SupersetTestCase
+from tests.integration_tests.constants import (
+ ADMIN_USERNAME,
+ ALPHA_USERNAME,
+ GAMMA_USERNAME,
+)
+from tests.integration_tests.fixtures.birth_names_dashboard import ( # noqa: F401
+ load_birth_names_dashboard_with_slices,
+ load_birth_names_data,
+)
+
+
+def _get_birth_names_dataset() -> SqlaTable:
+ return (
+ db.session.query(SqlaTable)
+ .filter(SqlaTable.table_name == "birth_names")
+ .first()
+ )
+
+
+def _persist_fixture_state() -> None:
+ """Force the fixture's pending INSERTs to commit so subsequent edits
+ produce *new* version rows instead of being batched into the
+ creation transaction. Mirrors the same helper in
+ ``tests/integration_tests/dashboards/version_history_tests.py``.
+ """
+ db.session.commit()
+
+
+def _get_birth_names_dashboard() -> Dashboard:
+ return (
+ db.session.query(Dashboard)
+ .filter(Dashboard.dashboard_title == "USA Births Names")
+ .first()
+ )
+
+
+class TestDashboardActivityView(SupersetTestCase):
+ """T017–T026 — ``GET /api/v1/dashboard/<uuid>/activity/`` (US1)."""
+
+ @pytest.fixture(autouse=True)
Review Comment:
Fixed in 8b19744b85 by adding the requested explicit type annotation.
##########
tests/integration_tests/versioning/change_records_tests.py:
##########
@@ -0,0 +1,715 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Integration tests for ``version_changes`` capture (T052, partial).
+
+Covers in this file:
+ (a) saving a chart with three field changes produces three rows
+ (f) baseline / INSERT transactions produce zero records *for that entity*
+ + unchanged-save / dashboard / params-classification cases
+
+Deferred:
+ (b) ``GET /versions/`` response includes ``changes`` array — lands with
+ T050 (API integration).
+ (c) FK cascade — exercisable in principle (the migration declares
+ ``ON DELETE CASCADE``) but can't be isolated in a unit-style test
+ because ``version_transaction`` is referenced by non-cascading FKs
+ from slices_version / dashboards_version / etc. Covered instead
+ by (d) below once it lands, and by the structural declaration in
+ T046's migration.
+ (d) retention prune drops change records alongside the pruned
+ version — will land when T049 extends ``VersionDAO.prune_versions``
+ to include ``version_changes`` alongside the shadow-row delete.
+ (e) ``kind`` index query plan on Postgres — deferred to T053 perf
+ validation.
+"""
+
+from __future__ import annotations
+
+from datetime import datetime, timedelta
+from typing import Any
+from uuid import uuid4
+
+import pytest
+import sqlalchemy as sa
+from sqlalchemy_continuum import version_class
+
+from superset.extensions import db
+from superset.models.dashboard import Dashboard
+from superset.models.slice import Slice
+from superset.utils import json as _json
+from tests.integration_tests.base_tests import SupersetTestCase
+from tests.integration_tests.fixtures.birth_names_dashboard import ( # noqa: F401
+ load_birth_names_dashboard_with_slices,
+ load_birth_names_data,
+)
+
+_VERSION_CHANGES = sa.table(
+ "version_changes",
+ sa.column("id"),
+ sa.column("transaction_id"),
+ sa.column("entity_kind"),
+ sa.column("entity_id"),
+ sa.column("sequence"),
+ sa.column("kind"),
+ sa.column("operation"),
+ sa.column("path"),
+ sa.column("from_value"),
+ sa.column("to_value"),
+)
+
+_VERSION_TRANSACTION = sa.table(
+ "version_transaction",
+ sa.column("id"),
+ sa.column("issued_at"),
+ sa.column("user_id"),
+ sa.column("action_kind"),
+)
+
+
+def _action_kind_for(tx_id: int) -> str | None:
+ """Read the ``action_kind`` column from the version_transaction row."""
+ return (
+ db.session.connection()
+ .execute(
+ sa.select(_VERSION_TRANSACTION.c.action_kind).where(
+ _VERSION_TRANSACTION.c.id == tx_id
+ )
+ )
+ .scalar()
+ )
+
+
+def _change_rows_for(
+ tx_id: int,
+ *,
+ entity_kind: str | None = None,
+ entity_id: int | None = None,
+) -> list[dict[str, Any]]:
+ """Raw fetch of ``version_changes`` rows for a tx + optional entity filter."""
+ query = sa.select(_VERSION_CHANGES).where(
+ _VERSION_CHANGES.c.transaction_id == tx_id
+ )
+ if entity_kind is not None:
+ query = query.where(_VERSION_CHANGES.c.entity_kind == entity_kind)
+ if entity_id is not None:
+ query = query.where(_VERSION_CHANGES.c.entity_id == entity_id)
+ query = query.order_by(_VERSION_CHANGES.c.sequence.asc())
+ result = db.session.connection().execute(query)
+ return [dict(row._mapping) for row in result]
+
+
+def _persist_fixture_state() -> None:
+ """Commit fixture INSERTs so the baseline row exists before the test edits.
+
+ Without this, the test's first commit batches the fixture's pending
+ INSERTs with the test's UPDATE into a single Continuum transaction
+ and no diff records are emitted (no pre-state).
+ """
+ db.session.commit()
+
+
+class TestChartChangeRecords(SupersetTestCase):
+ """Change-record capture for chart (Slice) saves."""
+
+ @pytest.fixture(autouse=True)
Review Comment:
Fixed in 8b19744b85 by adding the requested explicit type annotation.
##########
tests/integration_tests/versioning/perf_validation_tests.py:
##########
@@ -0,0 +1,416 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""T044 — Performance validation for entity version history.
+
+Skipped by default. Run on demand:
+
+ SUPERSET_PERF_VALIDATION=1 pytest \
+ tests/integration_tests/versioning/perf_validation_tests.py -v -s
+
+Measures the three success criteria defined in the spec:
+
+ * SC-002: version list endpoint responds in under 1 second
+ * SC-004: save path p95 overhead under 50 ms with Continuum tracking
+ on vs. off (FR-014)
+
+(SC-003, the restore-endpoint benchmark, lands with the restore endpoint
+in a later PR.)
+
+The test prints a summary table suitable for pasting into the PR
+description. It also asserts each target so regressions fail loudly
+when the harness is re-run.
+"""
+
+from __future__ import annotations
+
+import os
+import statistics
+import time
+from typing import Any
+
+import pytest
+import sqlalchemy as sa
+from sqlalchemy_continuum import version_class, versioning_manager
+
+from superset.extensions import db
+from superset.models.slice import Slice
+from tests.integration_tests.base_tests import SupersetTestCase
+from tests.integration_tests.constants import ADMIN_USERNAME
+from tests.integration_tests.fixtures.birth_names_dashboard import ( # noqa: F401
+ load_birth_names_dashboard_with_slices,
+ load_birth_names_data,
+)
+
+SKIP_REASON = "Performance validation is manual. Set SUPERSET_PERF_VALIDATION=1 to run."
+
+# Thresholds from spec.md §Success Criteria.
+LIST_ENDPOINT_MAX_MS = 1000 # SC-002
+SAVE_OVERHEAD_P95_MAX_MS = 50 # SC-004
+
+# Activity-view thresholds (sc-107283 §Success Criteria).
+ACTIVITY_ENDPOINT_P95_MAX_MS = 1500 # SC-AV-001
+
+
+def _save_chart_once(chart: Slice, suffix: str) -> None:
+ """One ORM-level save path, mimicking what ChartDAO.update does."""
+ chart.slice_name = f"{chart.slice_name[:64]}_{suffix}"
+ db.session.commit()
+
+
+def _timings_ms(seconds: list[float]) -> dict[str, float]:
+ ms = sorted(s * 1000.0 for s in seconds)
+ return {
+ "p50": statistics.median(ms),
+ "p95": ms[int(len(ms) * 0.95) - 1] if len(ms) >= 20 else max(ms),
+ "max": max(ms),
+ "n": len(ms),
+ }
+
+
+@pytest.mark.skipif(
+ not os.environ.get("SUPERSET_PERF_VALIDATION"),
+ reason=SKIP_REASON,
+)
+class PerfValidationTests(SupersetTestCase):
+ """Runs only when SUPERSET_PERF_VALIDATION=1 is set."""
+
+ @pytest.fixture(autouse=True)
+ def _load_data(self, load_birth_names_dashboard_with_slices: Any) -> None: # noqa: F811, PT004
+ pass
+
+ def _seed_chart_with_n_versions(self, n: int) -> Slice:
+ """Save a chart N times to produce N version rows."""
+ chart = db.session.query(Slice).first()
+ assert chart is not None, "birth_names fixture should provide charts"
+
+ for i in range(n):
+ _save_chart_once(chart, f"v{i}")
+ db.session.commit()
+ return chart
+
+ def test_sc002_list_endpoint_under_1s(self) -> None:
+ """SC-002: list endpoint responds in under 1 second."""
+ self.login(ADMIN_USERNAME)
+
+ # Generate enough versions to exercise the retention-capped state.
+ chart = self._seed_chart_with_n_versions(24)
+ chart_uuid = str(chart.uuid)
+ url = f"/api/v1/chart/{chart_uuid}/versions/"
+
+ # Warm up the endpoint once (JIT caching, mapper configuration, etc.)
+ self.client.get(url)
+
+ timings: list[float] = []
+ for _ in range(10):
+ t0 = time.perf_counter()
+ response = self.client.get(url)
+ timings.append(time.perf_counter() - t0)
+ assert response.status_code == 200
+
+ stats = _timings_ms(timings)
+ print(
+ f"\n[SC-002] GET /versions/ (24 versions) "
+ f"p50={stats['p50']:.1f}ms p95={stats['p95']:.1f}ms "
+ f"max={stats['max']:.1f}ms n={stats['n']}"
+ )
+ assert stats["p95"] < LIST_ENDPOINT_MAX_MS, (
+ f"SC-002 failed: list endpoint p95 {stats['p95']:.1f}ms "
+ f">= {LIST_ENDPOINT_MAX_MS}ms"
+ )
+
+ # SC-003 (restore endpoint under 3s) intentionally omitted here: the
+ # version-restore route (POST /<uuid>/versions/<version_uuid>/restore)
+ # ships in a later PR, so there is nothing to time at this point in the
+ # stack. Re-add this benchmark alongside the restore endpoint.
+
+ def test_sc004_save_overhead_under_50ms(self) -> None:
+ """SC-004: save path p95 overhead under 50ms (FR-014).
+
+ Toggling Continuum on and off mid-process corrupts its internal
+ ``units_of_work`` state and is not a reliable measurement. Instead
+ this test directly measures the wall-clock time spent inside the
+ four session-level listeners Continuum attaches to
+ ``sa.orm.session.Session`` — ``before_flush``, ``after_flush``,
+ ``after_commit``, ``after_rollback`` — plus Superset's own
+ baseline / snapshot / retention-prune listeners (attached to
+ ``db.session``). The cumulative listener time per save is the
+ marginal overhead version capture adds over a save with
+ versioning removed entirely, because without these listeners
+ the ORM would not execute any of that code.
+
+ The approach:
+ 1. Wrap each known listener with a timing proxy that adds its
+ wall-clock time to a per-save accumulator.
+ 2. Save the same chart N times, recording each save's
+ accumulator value.
+ 3. Compute p50 / p95 of the per-save overhead.
+
+ This matches the measurement intent of SC-004 (how much does
+ versioning cost per save) without the fragility of toggling
+ Continuum mid-test.
+ """
+ self.login(ADMIN_USERNAME)
+
+ chart = db.session.query(Slice).first()
+ assert chart is not None
+
+ # Per-save accumulator incremented by the wrapped listeners.
+ acc = [0.0]
Review Comment:
Addressed by subsequent commits; the flagged code is no longer current after the rebase onto master. Resolving as outdated.
##########
tests/integration_tests/versioning/perf_validation_tests.py:
##########
@@ -0,0 +1,416 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""T044 — Performance validation for entity version history.
+
+Skipped by default. Run on demand:
+
+ SUPERSET_PERF_VALIDATION=1 pytest \
+ tests/integration_tests/versioning/perf_validation_tests.py -v -s
+
+Measures the three success criteria defined in the spec:
+
+ * SC-002: version list endpoint responds in under 1 second
+ * SC-004: save path p95 overhead under 50 ms with Continuum tracking
+ on vs. off (FR-014)
+
+(SC-003, the restore-endpoint benchmark, lands with the restore endpoint
+in a later PR.)
+
+The test prints a summary table suitable for pasting into the PR
+description. It also asserts each target so regressions fail loudly
+when the harness is re-run.
+"""
+
+from __future__ import annotations
+
+import os
+import statistics
+import time
+from typing import Any
+
+import pytest
+import sqlalchemy as sa
+from sqlalchemy_continuum import version_class, versioning_manager
+
+from superset.extensions import db
+from superset.models.slice import Slice
+from tests.integration_tests.base_tests import SupersetTestCase
+from tests.integration_tests.constants import ADMIN_USERNAME
+from tests.integration_tests.fixtures.birth_names_dashboard import ( # noqa: F401
+ load_birth_names_dashboard_with_slices,
+ load_birth_names_data,
+)
+
+SKIP_REASON = "Performance validation is manual. Set SUPERSET_PERF_VALIDATION=1 to run."
+
+# Thresholds from spec.md §Success Criteria.
+LIST_ENDPOINT_MAX_MS = 1000 # SC-002
+SAVE_OVERHEAD_P95_MAX_MS = 50 # SC-004
+
+# Activity-view thresholds (sc-107283 §Success Criteria).
+ACTIVITY_ENDPOINT_P95_MAX_MS = 1500 # SC-AV-001
+
+
+def _save_chart_once(chart: Slice, suffix: str) -> None:
+ """One ORM-level save path, mimicking what ChartDAO.update does."""
+ chart.slice_name = f"{chart.slice_name[:64]}_{suffix}"
+ db.session.commit()
+
+
+def _timings_ms(seconds: list[float]) -> dict[str, float]:
+ ms = sorted(s * 1000.0 for s in seconds)
+ return {
+ "p50": statistics.median(ms),
+ "p95": ms[int(len(ms) * 0.95) - 1] if len(ms) >= 20 else max(ms),
+ "max": max(ms),
+ "n": len(ms),
+ }
+
+
+@pytest.mark.skipif(
+ not os.environ.get("SUPERSET_PERF_VALIDATION"),
+ reason=SKIP_REASON,
+)
+class PerfValidationTests(SupersetTestCase):
+ """Runs only when SUPERSET_PERF_VALIDATION=1 is set."""
+
+ @pytest.fixture(autouse=True)
+ def _load_data(self, load_birth_names_dashboard_with_slices: Any) -> None: # noqa: F811, PT004
+ pass
+
+ def _seed_chart_with_n_versions(self, n: int) -> Slice:
+ """Save a chart N times to produce N version rows."""
+ chart = db.session.query(Slice).first()
+ assert chart is not None, "birth_names fixture should provide charts"
+
+ for i in range(n):
+ _save_chart_once(chart, f"v{i}")
+ db.session.commit()
+ return chart
+
+ def test_sc002_list_endpoint_under_1s(self) -> None:
+ """SC-002: list endpoint responds in under 1 second."""
+ self.login(ADMIN_USERNAME)
+
+ # Generate enough versions to exercise the retention-capped state.
+ chart = self._seed_chart_with_n_versions(24)
+ chart_uuid = str(chart.uuid)
+ url = f"/api/v1/chart/{chart_uuid}/versions/"
+
+ # Warm up the endpoint once (JIT caching, mapper configuration, etc.)
+ self.client.get(url)
+
+ timings: list[float] = []
+ for _ in range(10):
+ t0 = time.perf_counter()
+ response = self.client.get(url)
+ timings.append(time.perf_counter() - t0)
+ assert response.status_code == 200
+
+ stats = _timings_ms(timings)
+ print(
+ f"\n[SC-002] GET /versions/ (24 versions) "
+ f"p50={stats['p50']:.1f}ms p95={stats['p95']:.1f}ms "
+ f"max={stats['max']:.1f}ms n={stats['n']}"
+ )
+ assert stats["p95"] < LIST_ENDPOINT_MAX_MS, (
+ f"SC-002 failed: list endpoint p95 {stats['p95']:.1f}ms "
+ f">= {LIST_ENDPOINT_MAX_MS}ms"
+ )
+
+ # SC-003 (restore endpoint under 3s) intentionally omitted here: the
+ # version-restore route (POST /<uuid>/versions/<version_uuid>/restore)
+ # ships in a later PR, so there is nothing to time at this point in the
+ # stack. Re-add this benchmark alongside the restore endpoint.
+
+ def test_sc004_save_overhead_under_50ms(self) -> None:
+ """SC-004: save path p95 overhead under 50ms (FR-014).
+
+ Toggling Continuum on and off mid-process corrupts its internal
+ ``units_of_work`` state and is not a reliable measurement. Instead
+ this test directly measures the wall-clock time spent inside the
+ four session-level listeners Continuum attaches to
+ ``sa.orm.session.Session`` — ``before_flush``, ``after_flush``,
+ ``after_commit``, ``after_rollback`` — plus Superset's own
+ baseline / snapshot / retention-prune listeners (attached to
+ ``db.session``). The cumulative listener time per save is the
+ marginal overhead version capture adds over a save with
+ versioning removed entirely, because without these listeners
+ the ORM would not execute any of that code.
+
+ The approach:
+ 1. Wrap each known listener with a timing proxy that adds its
+ wall-clock time to a per-save accumulator.
+ 2. Save the same chart N times, recording each save's
+ accumulator value.
+ 3. Compute p50 / p95 of the per-save overhead.
+
+ This matches the measurement intent of SC-004 (how much does
+ versioning cost per save) without the fragility of toggling
+ Continuum mid-test.
+ """
+ self.login(ADMIN_USERNAME)
+
+ chart = db.session.query(Slice).first()
+ assert chart is not None
+
+ # Per-save accumulator incremented by the wrapped listeners.
+ acc = [0.0]
+
+ def wrap_listener(original: Any) -> Any:
+ def wrapper(*args: Any, **kwargs: Any) -> Any:
+ t0 = time.perf_counter()
+ try:
+ return original(*args, **kwargs)
+ finally:
+ acc[0] += time.perf_counter() - t0
+
+ wrapper.__wrapped__ = original # type: ignore[attr-defined]
+ return wrapper
+
+ # Instrument Continuum's four session listeners by detaching the
+ # bound method, wrapping, and re-attaching under a single-use
+ # listener handle we can cleanly remove on teardown.
+ session_target = sa.orm.session.Session
+ attached: list[tuple[str, Any]] = []
+ for event_name, listener in list(versioning_manager.session_listeners.items()):
+ sa.event.remove(session_target, event_name, listener)
+ wrapped = wrap_listener(listener)
+ sa.event.listen(session_target, event_name, wrapped)
+ attached.append((event_name, wrapped))
+
+ iterations = 100
+ warmup = 5
+ try:
+ # Warmup (first baseline INSERT, JIT, cache warming).
+ for i in range(warmup):
+ _save_chart_once(chart, f"warm_{i}")
+ acc[0] = 0.0
+
+ total_timings: list[float] = []
+ overhead_timings: list[float] = []
+ for i in range(iterations):
+ acc[0] = 0.0
+ t0 = time.perf_counter()
+ _save_chart_once(chart, f"run_{i}")
+ total_timings.append(time.perf_counter() - t0)
+ overhead_timings.append(acc[0])
+ finally:
+ for event_name, wrapped in attached:
+ sa.event.remove(session_target, event_name, wrapped)
+ sa.event.listen(
+ session_target,
+ event_name,
+ wrapped.__wrapped__,
+ )
+
+ total = _timings_ms(total_timings)
+ overhead = _timings_ms(overhead_timings)
+
+ ver_cls = version_class(Slice)
+ produced = db.session.query(ver_cls).filter(ver_cls.id == chart.id).count()
+ print(
+ f"\n[SC-004] save iterations={iterations} chart_id={chart.id} "
+ f"version_rows_produced={produced}"
+ )
+ print(
+ f"[SC-004] full save: "
+ f"p50={total['p50']:.2f}ms p95={total['p95']:.2f}ms "
+ f"max={total['max']:.2f}ms"
+ )
+ print(
+ f"[SC-004] version-cap overhead: "
+ f"p50={overhead['p50']:.2f}ms p95={overhead['p95']:.2f}ms "
+ f"max={overhead['max']:.2f}ms"
+ )
+
+ assert overhead["p95"] < SAVE_OVERHEAD_P95_MAX_MS, (
+ f"SC-004 failed: version-capture p95 overhead "
+ f"{overhead['p95']:.2f}ms >= {SAVE_OVERHEAD_P95_MAX_MS}ms"
+ )
+
+ # ---- T045: Activity-view perf validation -----------------------------
+
+ def _seed_activity_history(self) -> str:
+ """Generate dense history on the birth_names dashboard so the
+ activity endpoint has something realistic to read.
+
+ T045's spec target is "25 charts × 3 dataset windows each". The
+ birth_names fixture has ~12 charts on a single dataset (no
+ multi-dataset support without a bespoke fixture). We approximate
+ the load by: (a) editing many charts on the dashboard, (b)
+ editing the dataset's description several times, (c) editing the
+ dashboard's own title once. That yields ~30+ change records
+ spanning all three entity kinds — enough to exercise the
+ decoration, visibility, and impact-batch paths without needing a
+ multi-dataset fixture builder. Returns the dashboard UUID.
+
+ **Why this commits without rollback** (unlike the test bodies in
+ ``activity_view_tests.py``): the whole point of a perf seed is
+ that the rows it produces have actually been persisted, so the
+ endpoint hit that follows reads a realistic state of the
+ ``version_changes`` / shadow tables. T053's
+ ``try/finally``+``rollback`` convention is for tests that
+ assert on *which records were captured*; here the seed IS the
+ setup, not the unit under test. The fixture's session-scoped
+ ``_cleanup`` removes the dashboard / slices at session teardown,
+ which is when the shadow rows age out too.
+ """
+ # pylint: disable=import-outside-toplevel
+ from superset.connectors.sqla.models import SqlaTable
+ from superset.models.dashboard import Dashboard
+
+ dashboard = (
+ db.session.query(Dashboard)
+ .filter(Dashboard.dashboard_title.like("USA Births%"))
+ .first()
+ )
Review Comment:
Fixed in 8b19744b85 by adding the requested explicit type annotation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(versioning): cross-entity version activity view [superset]
Posted by "codeant-ai-for-open-source[bot] (via GitHub)" <gi...@apache.org>.
codeant-ai-for-open-source[bot] commented on code in PR #41076:
URL: https://github.com/apache/superset/pull/41076#discussion_r3560196749
##########
superset/versioning/activity/impact.py:
##########
@@ -0,0 +1,186 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Per-record impact computation for the activity DTO.
+
+Only dashboard-path activity records pointing at a ``SqlaTable``
+related entity carry an ``impact`` field — the number of charts on
+the dashboard at that transaction that were pointing at the dataset.
+This module computes that count in a single batched query per
+request:
+
+* :func:`collect_impact_pairs` — pulls the distinct
+ ``(dataset_id, transaction_id)`` pairs that need counts.
+* :func:`batch_chart_counts` — one SQL query joining
+ ``dashboard_slices_version`` and ``slices_version`` to count
+ the matching charts validity-strategy-style.
+* :func:`impact_for_record` — pure projection from the pre-fetched
+ counts onto each record (returns ``None`` for non-Dashboard paths
+ or non-SqlaTable kinds, matching the ``impact`` computation).
+
+Splitting the count batching from the pure projection keeps the SQL
+inside one function (the batched read) and the per-record decoration
+inside another (no DB).
+"""
+
+from __future__ import annotations
+
+from typing import Any
+
+import sqlalchemy as sa
+
+from superset.extensions import db
+from superset.versioning.activity.kinds import (
+ chunked_ids,
+ ENTITY_ID_CHUNK_SIZE,
+ TABLE_KIND_TO_API,
+)
+
+
+def collect_impact_pairs(
+ records: list[dict[str, Any]], path_kind: str
+) -> set[tuple[int, int]]:
+ """Distinct ``(dataset_id, transaction_id)`` pairs from *records*
+ that require an impact computation.
+
+ Only dashboard-path records whose related entity is a ``SqlaTable``
+ produce a non-null ``impact`` field; for any other shape this set
+ is empty and no DB query needs to fire.
+ """
+ if path_kind != "Dashboard":
+ return set()
+ return {
+ (record["entity_id"], record["transaction_id"])
+ for record in records
+ if TABLE_KIND_TO_API.get(record["entity_kind"]) == "SqlaTable"
+ }
+
+
+def batch_chart_counts(
+ dashboard_id: int, pairs: set[tuple[int, int]]
+) -> dict[tuple[int, int], int]:
+ """For every ``(dataset_id, target_tx)`` in *pairs*, count the
+ distinct charts that were both on *dashboard_id* and pointing at
+ *dataset_id* at *target_tx*.
+
+ One SELECT against ``dashboard_slices_version`` ⨝ ``slices_version``,
+ pulling the (slice, dataset, validity-window) state for every slice
+ ever on the dashboard whose dataset matches one of the requested
+ dataset_ids. The Python loop then applies the validity-strategy
+ predicate per pair. Replaces the previous N+1 shape that fired one
+ COUNT per related record.
+
+ Returns ``{(dataset_id, target_tx): count}``; pairs whose count
+ would be zero are omitted so the caller's ``.get(key, 0)`` is
+ correct.
+ """
+ if not pairs:
+ return {}
+
+ # pylint: disable=import-outside-toplevel
+ from sqlalchemy_continuum import version_class
+
+ from superset.models.slice import Slice
+
+ metadata = version_class(Slice).__table__.metadata
+ m2m_tbl = metadata.tables.get("dashboard_slices_version")
+ slices_tbl = version_class(Slice).__table__
+ if m2m_tbl is None:
+ return {}
+
+ dataset_ids = {dataset_id for dataset_id, _ in pairs}
Review Comment:
**Suggestion:** Add an explicit type annotation for this computed collection to satisfy the requirement for annotating relevant variables. [custom_rule]
**Severity Level:** Minor 🧹
<details>
<summary><b>Why it matters? ⭐ </b></summary>
This is a newly added Python variable whose type is inferred rather than explicitly annotated. Under the stated rule, relevant variables that can be annotated should include type hints, so this suggestion matches a real violation.
</details>
<details>
<summary><b>Rule source 📖 </b></summary>
.cursor/rules/dev-standard.mdc (line 28)
</details>
[](https://app.codeant.ai/fix-in-ide?tool=cursor&prompt_id=d530900b515d46beaf998d8a3b590406&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset) [](https://app.codeant.ai/fix-in-ide?tool=vscode-claude&prompt_id=d530900b515d46beaf998d8a3b590406&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset)
*(Use Cmd/Ctrl + Click for best experience)*
<details>
<summary><b>Prompt for AI Agent 🤖 </b></summary>
```mdx
This is a comment left during a code review.
**Path:** superset/versioning/activity/impact.py
**Line:** 104:104
**Comment:**
*Custom Rule: Add an explicit type annotation for this computed collection to satisfy the requirement for annotating relevant variables.
Validate the correctness of the flagged issue. If correct, How can I resolve this? If you propose a fix, implement it and please make it concise.
Once fix is implemented, also check other comments on the same PR, and ask user if the user wants to fix the rest of the comments as well. if said yes, then fetch all the comments validate the correctness and implement a minimal fix
```
</details>
<a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F41076&comment_hash=298e20cc758a18d0ef496a756dccbf354ce3b3f3e8cfe237c3885865f860294b&reaction=like'>👍</a> | <a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F41076&comment_hash=298e20cc758a18d0ef496a756dccbf354ce3b3f3e8cfe237c3885865f860294b&reaction=dislike'>👎</a>
##########
superset/versioning/activity/__init__.py:
##########
@@ -0,0 +1,102 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Read-side queries for the cross-entity activity-view API.
+
+Companion to :mod:`superset.versioning.queries`. Whereas ``queries.py``
+returns transaction-level history for a single entity, the helpers
+here unify change-record history across an entity's transitive
+dependency chain — a dashboard's activity stream includes edits to
+charts that were attached to it AND edits to datasets those charts
+pointed at, each time-bounded by when the relationship was active.
+
+One public entry point — :func:`get_activity` — dispatches on the
+first argument to serve all three endpoint families:
+
+* ``get_activity(Dashboard, dashboard_uuid, ...)`` — own edits +
+ charts attached during their dashboard window + datasets those
+ charts used during their chart window.
+* ``get_activity(Slice, chart_uuid, ...)`` — own edits + datasets the
+ chart pointed at during association.
+* ``get_activity(SqlaTable, dataset_uuid, ...)`` — own edits only.
+ Datasets are not transitive recipients of activity in V2.
+
+Package layout (descends from public entry point to leaf helpers):
+
+* :mod:`.orchestrator` — :func:`get_activity` (public), the
+ ``activity_endpoint`` REST helper, the request param parser
+ (:func:`parse_activity_query_params`), and the observability
+ instrumentation (request-shape + per-kind metrics).
+* :mod:`.scope` — scope resolution (DB-touching):
+ :func:`resolve_scope` / :func:`_resolve_dashboard_scope` /
+ :func:`_resolve_chart_scope` / :func:`_resolve_related_scope`.
+* :mod:`.windows` — pure window arithmetic on half-open
+ ``[start_tx, end_tx)`` intervals: :func:`intersect_windows` /
+ :func:`union_windows` / :func:`merge_entity_windows` /
+ :func:`row_within_any_window`. Extracted from :mod:`.scope` so
+ :mod:`.queries` can import the pure helpers at module-top instead
+ of through a cycle-dodging lazy import.
+* :mod:`.queries` — every DB-touching helper: Phase A relationship
+ walks, Phase B change-record fetch, name denormalization,
+ path-entity resolution, and tombstone-state lookup.
+* :mod:`.impact` — per-record impact-count computation (the only
+ field that requires its own batched query).
+* :mod:`.visibility` — the silent visibility filter; uses
+ the same SQL-side access filters FAB applies on list endpoints.
+* :mod:`.render` — record-decoration helpers that turn raw rows into
+ the ActivityRecord DTO (summary headlines, ``changed_by`` projection,
+ uuid lookup).
+* :mod:`.kinds` — the kind-translation tables, the ``Window`` /
+ ``EntityWindows`` type aliases, and :func:`load_live_model`.
+
+The public surface (re-exported here) is the eight symbols below.
+Sub-module privates are intentionally NOT re-exported — tests and
+new internal callers should import them from their owning submodule
+(e.g. ``from superset.versioning.activity.windows import
+intersect_windows``) so the package's public API stays scannable.
+
+``PathEntityResponseError`` and ``resolve_endpoint_path_entity`` are
+re-exported here from :mod:`superset.versioning.api_helpers` (where
+they live alongside the ``/versions/`` endpoint handlers) so the
+three ``/activity/`` endpoint callers can ``from
+superset.versioning.activity import resolve_endpoint_path_entity``
+without crossing into the ``/versions/`` module name.
+"""
+
+from __future__ import annotations
+
+from superset.versioning.activity.kinds import EntityWindows, Window
+from superset.versioning.activity.orchestrator import (
+ activity_endpoint,
+ ActivityParamsError,
+ get_activity,
+ parse_activity_query_params,
+)
+from superset.versioning.api_helpers import (
+ PathEntityResponseError,
+ resolve_endpoint_path_entity,
+)
+
+__all__ = [
+ "ActivityParamsError",
+ "EntityWindows",
+ "PathEntityResponseError",
+ "Window",
+ "activity_endpoint",
+ "get_activity",
+ "parse_activity_query_params",
+ "resolve_endpoint_path_entity",
+]
Review Comment:
**Suggestion:** Add an explicit variable type annotation for the exported symbol list so it is covered by static type checking. [custom_rule]
**Severity Level:** Minor 🧹
<details>
<summary><b>Why it matters? ⭐ </b></summary>
The rule requires type hints on relevant variables that can be annotated. `__all__` is a module-level variable with a clear type (`list[str]`) and is currently unannotated, so this is a real violation.
</details>
<details>
<summary><b>Rule source 📖 </b></summary>
.cursor/rules/dev-standard.mdc (line 28)
</details>
[](https://app.codeant.ai/fix-in-ide?tool=cursor&prompt_id=3e932ca1378e4e7dbe54a90cba173476&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset) [](https://app.codeant.ai/fix-in-ide?tool=vscode-claude&prompt_id=3e932ca1378e4e7dbe54a90cba173476&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset)
*(Use Cmd/Ctrl + Click for best experience)*
<details>
<summary><b>Prompt for AI Agent 🤖 </b></summary>
```mdx
This is a comment left during a code review.
**Path:** superset/versioning/activity/__init__.py
**Line:** 93:102
**Comment:**
*Custom Rule: Add an explicit variable type annotation for the exported symbol list so it is covered by static type checking.
Validate the correctness of the flagged issue. If correct, How can I resolve this? If you propose a fix, implement it and please make it concise.
Once fix is implemented, also check other comments on the same PR, and ask user if the user wants to fix the rest of the comments as well. if said yes, then fetch all the comments validate the correctness and implement a minimal fix
```
</details>
<a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F41076&comment_hash=14f99bd32ee3c2c71165f9cb0d77bcc687697bb4cd4b4e787f03ccd7b6fb0138&reaction=like'>👍</a> | <a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F41076&comment_hash=14f99bd32ee3c2c71165f9cb0d77bcc687697bb4cd4b4e787f03ccd7b6fb0138&reaction=dislike'>👎</a>
##########
superset/versioning/activity/impact.py:
##########
@@ -0,0 +1,186 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Per-record impact computation for the activity DTO.
+
+Only dashboard-path activity records pointing at a ``SqlaTable``
+related entity carry an ``impact`` field — the number of charts on
+the dashboard at that transaction that were pointing at the dataset.
+This module computes that count in a single batched query per
+request:
+
+* :func:`collect_impact_pairs` — pulls the distinct
+ ``(dataset_id, transaction_id)`` pairs that need counts.
+* :func:`batch_chart_counts` — one SQL query joining
+ ``dashboard_slices_version`` and ``slices_version`` to count
+ the matching charts validity-strategy-style.
+* :func:`impact_for_record` — pure projection from the pre-fetched
+ counts onto each record (returns ``None`` for non-Dashboard paths
+ or non-SqlaTable kinds, matching the ``impact`` computation).
+
+Splitting the count batching from the pure projection keeps the SQL
+inside one function (the batched read) and the per-record decoration
+inside another (no DB).
+"""
+
+from __future__ import annotations
+
+from typing import Any
+
+import sqlalchemy as sa
+
+from superset.extensions import db
+from superset.versioning.activity.kinds import (
+ chunked_ids,
+ ENTITY_ID_CHUNK_SIZE,
+ TABLE_KIND_TO_API,
+)
+
+
+def collect_impact_pairs(
+ records: list[dict[str, Any]], path_kind: str
+) -> set[tuple[int, int]]:
+ """Distinct ``(dataset_id, transaction_id)`` pairs from *records*
+ that require an impact computation.
+
+ Only dashboard-path records whose related entity is a ``SqlaTable``
+ produce a non-null ``impact`` field; for any other shape this set
+ is empty and no DB query needs to fire.
+ """
+ if path_kind != "Dashboard":
+ return set()
+ return {
+ (record["entity_id"], record["transaction_id"])
+ for record in records
+ if TABLE_KIND_TO_API.get(record["entity_kind"]) == "SqlaTable"
+ }
+
+
+def batch_chart_counts(
+ dashboard_id: int, pairs: set[tuple[int, int]]
+) -> dict[tuple[int, int], int]:
+ """For every ``(dataset_id, target_tx)`` in *pairs*, count the
+ distinct charts that were both on *dashboard_id* and pointing at
+ *dataset_id* at *target_tx*.
+
+ One SELECT against ``dashboard_slices_version`` ⨝ ``slices_version``,
+ pulling the (slice, dataset, validity-window) state for every slice
+ ever on the dashboard whose dataset matches one of the requested
+ dataset_ids. The Python loop then applies the validity-strategy
+ predicate per pair. Replaces the previous N+1 shape that fired one
+ COUNT per related record.
+
+ Returns ``{(dataset_id, target_tx): count}``; pairs whose count
+ would be zero are omitted so the caller's ``.get(key, 0)`` is
+ correct.
+ """
+ if not pairs:
+ return {}
+
+ # pylint: disable=import-outside-toplevel
+ from sqlalchemy_continuum import version_class
+
+ from superset.models.slice import Slice
+
+ metadata = version_class(Slice).__table__.metadata
+ m2m_tbl = metadata.tables.get("dashboard_slices_version")
+ slices_tbl = version_class(Slice).__table__
+ if m2m_tbl is None:
+ return {}
+
+ dataset_ids = {dataset_id for dataset_id, _ in pairs}
+ # Bound both validity windows to the transaction range the page-set
+ # needs. Both the attachment (m2m) and the chart→dataset (slice) window
+ # must straddle a requested target_tx, so a row whose window starts
+ # after the newest target, or closes at/before the oldest one, can
+ # never contribute a match. Without this the join multiplies every
+ # attachment row by the full slice version history — an unbounded cross
+ # product on dashboards with long-lived, frequently-edited charts.
+ target_txs = {target_tx for _, target_tx in pairs}
Review Comment:
**Suggestion:** Add an explicit type annotation for this transaction-id set so the variable is fully typed rather than inferred. [custom_rule]
**Severity Level:** Minor 🧹
<details>
<summary><b>Why it matters? ⭐ </b></summary>
This is also a newly added inferred local variable in Python without an explicit annotation. The custom rule requires type hints for relevant variables that can be annotated, so this is a verified violation.
</details>
<details>
<summary><b>Rule source 📖 </b></summary>
.cursor/rules/dev-standard.mdc (line 28)
</details>
[](https://app.codeant.ai/fix-in-ide?tool=cursor&prompt_id=6e21c67cd564492598f2a844868789c7&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset) [](https://app.codeant.ai/fix-in-ide?tool=vscode-claude&prompt_id=6e21c67cd564492598f2a844868789c7&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset)
*(Use Cmd/Ctrl + Click for best experience)*
<details>
<summary><b>Prompt for AI Agent 🤖 </b></summary>
```mdx
This is a comment left during a code review.
**Path:** superset/versioning/activity/impact.py
**Line:** 112:112
**Comment:**
*Custom Rule: Add an explicit type annotation for this transaction-id set so the variable is fully typed rather than inferred.
Validate the correctness of the flagged issue. If correct, How can I resolve this? If you propose a fix, implement it and please make it concise.
Once fix is implemented, also check other comments on the same PR, and ask user if the user wants to fix the rest of the comments as well. if said yes, then fetch all the comments validate the correctness and implement a minimal fix
```
</details>
<a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F41076&comment_hash=dec93b730e49b3ef6c0f063b03a59b3e9626ca230e429631ba1ea2e054398940&reaction=like'>👍</a> | <a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F41076&comment_hash=dec93b730e49b3ef6c0f063b03a59b3e9626ca230e429631ba1ea2e054398940&reaction=dislike'>👎</a>
##########
superset/versioning/activity/scope.py:
##########
@@ -0,0 +1,118 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Scope resolution — turn a path entity into the related-entity walk.
+
+Composes :mod:`~superset.versioning.activity.queries` (Phase A
+relationship walks) and :mod:`~superset.versioning.activity.windows`
+(pure interval arithmetic) into the
+``list[EntityWindows]`` scope that
+:func:`~superset.versioning.activity.queries.fetch_change_records`
+consumes.
+
+The functions here read the DB (via the Phase A helpers in
+:mod:`~superset.versioning.activity.queries`); the pure window-
+arithmetic functions previously colocated here now live in
+:mod:`~superset.versioning.activity.windows` so the package no longer
+needs a lazy import to dodge a ``scope ↔ queries`` cycle.
+"""
+
+from __future__ import annotations
+
+from superset.versioning.activity.kinds import EntityWindows, Window
+from superset.versioning.activity.queries import (
+ batch_datasets_used_by_charts,
+ charts_attached_to_dashboard,
+ datasets_used_by_chart,
+)
+from superset.versioning.activity.windows import (
+ intersect_windows,
+ merge_entity_windows,
+)
+
+
+def resolve_scope(
+ path_kind: str,
+ path_id: int,
+ include: str,
+ self_start_tx: int | None = None,
+) -> list[EntityWindows]:
+ """Build the ``[(api_kind, entity_id, [windows])]`` list that
+ :func:`~superset.versioning.activity.queries.fetch_change_records`
+ consumes, branching by *path_kind* and *include* mode.
+
+ *self_start_tx* lower-bounds the self window at the path entity's first
+ tracked transaction (see
+ :func:`~superset.versioning.activity.queries.first_tracked_tx`). This
+ scopes the self stream to the current entity's own history: matching
+ self records on the bare integer id would otherwise inherit a
+ hard-deleted predecessor's change records under id reuse
+ (SQLite/MySQL reuse ``max(id)+1``). ``None`` means the entity has no
+ tracked history yet, so the self stream contributes nothing.
+ """
+ want_self = include in ("all", "self")
Review Comment:
**Suggestion:** Add an explicit boolean type annotation to this local control variable to satisfy the required type-hinting rule. [custom_rule]
**Severity Level:** Minor 🧹
<details>
<summary><b>Why it matters? ⭐ </b></summary>
The rule requires type hints on relevant variables that can be annotated. This local boolean flag is newly added, clearly inferable as `bool`, and is unannotated in the final code, so the suggestion matches a real violation.
</details>
<details>
<summary><b>Rule source 📖 </b></summary>
.cursor/rules/dev-standard.mdc (line 28)
</details>
[](https://app.codeant.ai/fix-in-ide?tool=cursor&prompt_id=9499b5f7bf494cb1a0d444cb0004417d&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset) [](https://app.codeant.ai/fix-in-ide?tool=vscode-claude&prompt_id=9499b5f7bf494cb1a0d444cb0004417d&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset)
*(Use Cmd/Ctrl + Click for best experience)*
<details>
<summary><b>Prompt for AI Agent 🤖 </b></summary>
```mdx
This is a comment left during a code review.
**Path:** superset/versioning/activity/scope.py
**Line:** 66:66
**Comment:**
*Custom Rule: Add an explicit boolean type annotation to this local control variable to satisfy the required type-hinting rule.
Validate the correctness of the flagged issue. If correct, How can I resolve this? If you propose a fix, implement it and please make it concise.
Once fix is implemented, also check other comments on the same PR, and ask user if the user wants to fix the rest of the comments as well. if said yes, then fetch all the comments validate the correctness and implement a minimal fix
```
</details>
<a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F41076&comment_hash=e6ab8d2bc175f01cc929c10d51e4cb503a90445b62164db00f89e0d40017c49d&reaction=like'>👍</a> | <a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F41076&comment_hash=e6ab8d2bc175f01cc929c10d51e4cb503a90445b62164db00f89e0d40017c49d&reaction=dislike'>👎</a>
##########
superset/versioning/activity/scope.py:
##########
@@ -0,0 +1,118 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Scope resolution — turn a path entity into the related-entity walk.
+
+Composes :mod:`~superset.versioning.activity.queries` (Phase A
+relationship walks) and :mod:`~superset.versioning.activity.windows`
+(pure interval arithmetic) into the
+``list[EntityWindows]`` scope that
+:func:`~superset.versioning.activity.queries.fetch_change_records`
+consumes.
+
+The functions here read the DB (via the Phase A helpers in
+:mod:`~superset.versioning.activity.queries`); the pure window-
+arithmetic functions previously colocated here now live in
+:mod:`~superset.versioning.activity.windows` so the package no longer
+needs a lazy import to dodge a ``scope ↔ queries`` cycle.
+"""
+
+from __future__ import annotations
+
+from superset.versioning.activity.kinds import EntityWindows, Window
+from superset.versioning.activity.queries import (
+ batch_datasets_used_by_charts,
+ charts_attached_to_dashboard,
+ datasets_used_by_chart,
+)
+from superset.versioning.activity.windows import (
+ intersect_windows,
+ merge_entity_windows,
+)
+
+
+def resolve_scope(
+ path_kind: str,
+ path_id: int,
+ include: str,
+ self_start_tx: int | None = None,
+) -> list[EntityWindows]:
+ """Build the ``[(api_kind, entity_id, [windows])]`` list that
+ :func:`~superset.versioning.activity.queries.fetch_change_records`
+ consumes, branching by *path_kind* and *include* mode.
+
+ *self_start_tx* lower-bounds the self window at the path entity's first
+ tracked transaction (see
+ :func:`~superset.versioning.activity.queries.first_tracked_tx`). This
+ scopes the self stream to the current entity's own history: matching
+ self records on the bare integer id would otherwise inherit a
+ hard-deleted predecessor's change records under id reuse
+ (SQLite/MySQL reuse ``max(id)+1``). ``None`` means the entity has no
+ tracked history yet, so the self stream contributes nothing.
+ """
+ want_self = include in ("all", "self")
+ want_related = include in ("all", "related")
+
+ scope: list[EntityWindows] = []
+ if want_self and self_start_tx is not None:
+ scope.append((path_kind, path_id, [Window(self_start_tx, None)]))
+ if want_related:
+ scope.extend(_resolve_related_scope(path_kind, path_id))
+ return scope
+
+
+def _resolve_related_scope(path_kind: str, path_id: int) -> list[EntityWindows]:
+ """Walk the dependency edges from the path entity to its related
+ entities. Datasets have no transitive layer in V2."""
+ if path_kind == "Dashboard":
+ return _resolve_dashboard_scope(path_id)
+ if path_kind == "Slice":
+ return _resolve_chart_scope(path_id)
+ return []
+
+
+def _resolve_dashboard_scope(dashboard_id: int) -> list[EntityWindows]:
+ """Charts on the dashboard during their attachment window, plus
+ datasets each chart pointed at during the intersection of (chart-
+ attachment, chart-on-dataset)."""
+ scope: list[EntityWindows] = []
+ chart_windows: dict[int, list[Window]] = {}
+ for slice_id, window in charts_attached_to_dashboard(dashboard_id):
+ chart_windows.setdefault(slice_id, []).append(window)
+
+ # One query for the dataset-history of every chart on the dashboard,
+ # not one query per chart. The per-slice form was O(n_charts) round-
+ # trips which dominated p95 on rich dashboards.
+ dataset_windows_by_slice = batch_datasets_used_by_charts(set(chart_windows))
Review Comment:
**Suggestion:** Add an explicit type annotation for this mapping variable so its key/value structure is declared at assignment. [custom_rule]
**Severity Level:** Minor 🧹
<details>
<summary><b>Why it matters? ⭐ </b></summary>
This local collection is inferred from the return value and is used as a mapping immediately after. Under the stated rule, it is a relevant variable that can be annotated but currently lacks an explicit type hint.
</details>
<details>
<summary><b>Rule source 📖 </b></summary>
.cursor/rules/dev-standard.mdc (line 28)
</details>
[](https://app.codeant.ai/fix-in-ide?tool=cursor&prompt_id=02fda2be19094382b8145dd580cf646d&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset) [](https://app.codeant.ai/fix-in-ide?tool=vscode-claude&prompt_id=02fda2be19094382b8145dd580cf646d&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset)
*(Use Cmd/Ctrl + Click for best experience)*
<details>
<summary><b>Prompt for AI Agent 🤖 </b></summary>
```mdx
This is a comment left during a code review.
**Path:** superset/versioning/activity/scope.py
**Line:** 99:99
**Comment:**
*Custom Rule: Add an explicit type annotation for this mapping variable so its key/value structure is declared at assignment.
Validate the correctness of the flagged issue. If correct, How can I resolve this? If you propose a fix, implement it and please make it concise.
Once fix is implemented, also check other comments on the same PR, and ask user if the user wants to fix the rest of the comments as well. if said yes, then fetch all the comments validate the correctness and implement a minimal fix
```
</details>
<a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F41076&comment_hash=bff4d447f2659c0dffb134d643e9eea84f7a5b7e874c4c5f44d03c7407246b10&reaction=like'>👍</a> | <a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F41076&comment_hash=bff4d447f2659c0dffb134d643e9eea84f7a5b7e874c4c5f44d03c7407246b10&reaction=dislike'>👎</a>
##########
superset/versioning/activity/queries.py:
##########
@@ -0,0 +1,754 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""DB-touching helpers for the activity-view read path.
+
+All Phase A relationship walks (``charts_attached_to_dashboard``,
+``datasets_used_by_chart``, ``batch_datasets_used_by_charts``),
+the Phase B change-record fetch (``fetch_change_records`` /
+``_select_change_rows_for_kinds``), the name-denormalization helpers
+(``_resolve_names_for_kind`` / ``apply_entity_name_denormalization``), the
+path-entity resolution helper (``resolve_path_entity``), and the
+tombstone-state lookup (``check_entity_tombstones``) live here.
+
+Each helper is a thin SELECT-and-shape function — no orchestration,
+no business logic. Callers in :mod:`scope`, :mod:`render`, and
+:mod:`orchestrator` compose them into the end-to-end request.
+
+**Inline imports.** Continuum's ``version_class`` / ``versioning_manager``
+and the Superset model classes are imported inside each helper because
+this package is loaded from ``init_versioning()`` before all SQLAlchemy
+mappers are configured.
+"""
+
+from __future__ import annotations
+
+import logging
+from datetime import datetime
+from typing import Any
+from uuid import UUID
+
+import sqlalchemy as sa
+from flask_appbuilder import Model
+
+from superset.extensions import db
+from superset.versioning.activity.kinds import (
+ API_KIND_TO_TABLE,
+ chunked_ids,
+ EntityWindows,
+ load_live_model,
+ NAME_COLUMN,
+ NOT_FOUND_EXC,
+ TABLE_KIND_TO_API,
+ Window,
+)
+from superset.versioning.activity.windows import row_within_any_window
+from superset.versioning.changes import version_changes_table
+
+logger = logging.getLogger(__name__)
+
+# ---- Path-entity resolution -----------------------------------------------
+
+
+def resolve_path_entity(model_cls: type[Model], entity_uuid: UUID) -> tuple[Any, int]:
+ """Resolve *entity_uuid* to ``(live_entity, entity_id)`` or raise a
+ typed 404.
+
+ Soft-delete handling is inherited transparently from
+ :func:`superset.versioning.queries.find_active_by_uuid` once it
+ learns to filter out ``deleted_at IS NOT NULL`` rows; at that point
+ soft-deleted paths will also raise here.
+ """
+ # pylint: disable=import-outside-toplevel
+ from superset.versioning.queries import find_active_by_uuid
+
+ entity = find_active_by_uuid(model_cls, entity_uuid)
+ if entity is None:
+ api_kind = model_cls.__name__
+ exc_cls = NOT_FOUND_EXC.get(api_kind)
+ if exc_cls is None:
+ raise LookupError(
+ f"Activity view does not support model class {api_kind!r}"
+ )
+ raise exc_cls(str(entity_uuid))
+ return entity, entity.id
+
+
+def first_tracked_tx(
+ model_cls: type[Model], entity_id: int, entity_uuid: UUID
+) -> int | None:
+ """Return the earliest Continuum transaction_id of the shadow rows
+ that belong to *this* live entity, matched on ``(id, uuid)``; ``None``
+ when the entity has no tracked history yet.
+
+ Used to lower-bound the self-scope window. Matching on the integer
+ ``id`` alone would inherit a previously hard-deleted entity's history
+ under id reuse (SQLite/MySQL reuse ``max(id)+1``); pinning the uuid
+ too — the same discrimination :func:`mark_first_tracked_saves` uses —
+ scopes the window to the current entity's own transactions.
+ """
+ # pylint: disable=import-outside-toplevel
+ from sqlalchemy_continuum import version_class
+
+ shadow_tbl = version_class(model_cls).__table__
+ stmt = sa.select(sa.func.min(shadow_tbl.c.transaction_id)).where(
+ shadow_tbl.c.id == entity_id,
+ shadow_tbl.c.uuid == entity_uuid,
+ )
+ return db.session.connection().execute(stmt).scalar()
+
+
+# ---- Phase A: relationship-traversal queries ------------------------------
+
+
+def charts_attached_to_dashboard(dashboard_id: int) -> list[tuple[int, Window]]:
+ """Return ``(slice_id, window)`` for every chart that has ever been on
+ *dashboard_id*, with each association's validity window in
+ transaction-id space.
+
+ Reads from ``dashboard_slices_version`` (Continuum's auto-generated
+ M2M shadow). Rows with ``operation_type = 2`` (DELETE) are excluded
+ so we don't synthesize a phantom window from a detachment row.
+ """
+ # pylint: disable=import-outside-toplevel
+ from sqlalchemy_continuum import version_class
+
+ from superset.models.dashboard import Dashboard
+
+ metadata = version_class(Dashboard).__table__.metadata
+ m2m_tbl = metadata.tables.get("dashboard_slices_version")
+ if m2m_tbl is None:
+ return []
+
+ rows = (
+ db.session.connection()
+ .execute(
+ sa.select(
+ m2m_tbl.c.slice_id,
+ m2m_tbl.c.transaction_id,
+ m2m_tbl.c.end_transaction_id,
+ ).where(
+ m2m_tbl.c.dashboard_id == dashboard_id,
+ m2m_tbl.c.operation_type != 2,
+ m2m_tbl.c.slice_id.is_not(None),
+ )
+ )
+ .all()
+ )
+ result: list[tuple[int, Window]] = []
+ for row in rows:
+ try:
+ window = Window(row[1], row[2])
+ except ValueError:
+ # A degenerate shadow row (end_tx <= start_tx) must not 500 the
+ # endpoint; skip it and leave a breadcrumb for investigation.
+ logger.warning(
+ "activity: skipping degenerate dashboard_slices_version row "
+ "(dashboard_id=%s, slice_id=%s, tx=%s, end_tx=%s)",
+ dashboard_id,
+ row[0],
+ row[1],
+ row[2],
+ )
+ continue
+ result.append((row[0], window))
+ return result
+
+
+def datasets_used_by_chart(slice_id: int) -> list[tuple[int, Window]]:
+ """Return ``(datasource_id, window)`` for every dataset that *slice_id*
+ has ever pointed at, with each association's validity window.
+
+ Single-slice form, used by ``_resolve_chart_scope`` where there
+ is only one chart to walk. The dashboard-scope path calls
+ :func:`batch_datasets_used_by_charts` instead so the query fires
+ once for all slices on the dashboard, not once per slice.
+
+ Reads from ``slices_version`` (the chart parent shadow). Filters to
+ ``datasource_type = 'table'`` because the activity view only follows
+ the chart → ``SqlaTable`` dependency edge (not legacy/other
+ datasources). Rows with ``operation_type = 2`` are excluded.
+ """
+ return batch_datasets_used_by_charts({slice_id}).get(slice_id, [])
+
+
+def batch_datasets_used_by_charts(
+ slice_ids: set[int],
+) -> dict[int, list[tuple[int, Window]]]:
+ """Batch form of :func:`datasets_used_by_chart`. Returns
+ ``{slice_id: [(dataset_id, window), ...]}`` in a single query so the
+ dashboard-scope walker doesn't fire one query per chart on the
+ dashboard. The previous per-slice shape became O(n_charts) round-
+ trips, which dominated ``get_activity`` latency on dashboards with
+ rich history (profile run 2026-05-26 showed `resolve_scope`
+ accounting for ~1.9s out of 4s p95).
+ """
+ if not slice_ids:
+ return {}
+
+ # pylint: disable=import-outside-toplevel
+ from sqlalchemy_continuum import version_class
+
+ from superset.models.slice import Slice
+
+ slices_tbl = version_class(Slice).__table__
+ grouped: dict[int, list[tuple[int, Window]]] = {}
+ # Chunk the IN-clause under SQLite's bind-variable floor (a dashboard can
+ # carry more charts than the floor allows in one statement).
+ rows: list[Any] = []
+ for chunk in chunked_ids(slice_ids):
+ rows.extend(
+ db.session.connection()
+ .execute(
+ sa.select(
+ slices_tbl.c.id,
+ slices_tbl.c.datasource_id,
+ slices_tbl.c.transaction_id,
+ slices_tbl.c.end_transaction_id,
+ ).where(
+ slices_tbl.c.id.in_(chunk),
+ slices_tbl.c.datasource_type == "table",
+ slices_tbl.c.operation_type != 2,
+ slices_tbl.c.datasource_id.is_not(None),
+ )
+ )
+ .mappings()
+ .all()
+ )
+ for row in rows:
+ try:
+ window = Window(row["transaction_id"], row["end_transaction_id"])
+ except ValueError:
+ # A degenerate shadow row (end_tx <= start_tx) must not 500 the
+ # endpoint; skip it and leave a breadcrumb for investigation.
+ logger.warning(
+ "activity: skipping degenerate slices_version row "
+ "(slice_id=%s, datasource_id=%s, tx=%s, end_tx=%s)",
+ row["id"],
+ row["datasource_id"],
+ row["transaction_id"],
+ row["end_transaction_id"],
+ )
+ continue
+ grouped.setdefault(row["id"], []).append((row["datasource_id"], window))
+ return grouped
+
+
+# ---- Phase B: change-record fetch -----------------------------------------
+
+
+def fetch_change_records(
+ entity_window_tuples: list[EntityWindows],
+ since: datetime | None,
+ until: datetime | None,
+) -> tuple[list[dict[str, Any]], bool]:
+ """Fetch all ``version_changes`` rows matching any of the supplied
+ entity-window tuples, joined with ``version_transaction`` for
+ ``issued_at`` and ``user_id``.
+
+ Each tuple is ``(api_kind, entity_id, [(start_tx, end_tx), ...])``;
+ a record matches when ``entity_kind`` equals the table-stored form
+ of *api_kind*, ``entity_id`` matches, and ``transaction_id`` falls
+ inside at least one of the entity's windows. ``since``/``until``
+ further restrict by ``issued_at``.
+
+ Implementation: one SELECT per kind with ``entity_id IN (...)`` and
+ a wide ``transaction_id`` bound (the union of all windows for that
+ kind). Per-window precision is applied in Python afterward. This
+ keeps the SQL shape proportional to the number of *kinds* (≤3) and
+ the bound proportional to the union of windows, not the cross-
+ product of (entity, window) — which previously generated one OR
+ clause per (entity, window) pair and hit SQLite's
+ ``SQLITE_MAX_EXPR_DEPTH`` limit on dashboards with many slices
+ or many historical attachment windows.
+
+ The visibility filter runs after this function (records
+ the requester can't read are silently dropped and must not
+ contribute to ``count``), so the orchestrator paginates in Python
+ over the filtered list — there is no DB-level page ``OFFSET`` here.
+ There IS a per-kind safety ceiling (``_MAX_FETCHED_RECORDS``) that
+ bounds how much history a single request materializes (to at most
+ ``n_kinds * _MAX_FETCHED_RECORDS``); when a kind exhausts it, the
+ second return value is ``True``, the returned stream is clamped to a
+ clean time cut (records older than the highest per-kind fetch floor
+ are dropped so truncation is a time boundary, not per-kind holes), and
+ the caller surfaces ``truncated`` on the response.
+
+ Returns ``(records, truncated)``. Records are ordered by
+ ``(issued_at DESC, transaction_id DESC, sequence DESC)`` — the
+ secondary keys break ties for the stable-ordering contract.
+ """
+ if not entity_window_tuples:
+ return [], False
+
+ # Group windows by (table_kind, entity_id) and by table_kind for SQL
+ # narrowing. The fetch is per-kind; the post-filter is per-entity.
+ windows_by_entity: dict[tuple[str, int], list[Window]] = {}
+ ids_by_kind: dict[str, set[int]] = {}
+ for api_kind, entity_id, windows in entity_window_tuples:
+ table_kind = API_KIND_TO_TABLE.get(api_kind)
+ if table_kind is None or not windows:
+ continue
+ ids_by_kind.setdefault(table_kind, set()).add(entity_id)
+ windows_by_entity.setdefault((table_kind, entity_id), []).extend(windows)
+
+ if not ids_by_kind:
+ return [], False
+
+ # Per-kind transaction_id bounds = the union of that kind's windows.
+ # Pushing these into the SQL WHERE ensures the per-statement LIMIT
+ # selects from IN-WINDOW rows. Without it, a related entity whose
+ # association window is far in the past would have the newest ``limit``
+ # (out-of-window) rows fetched and discarded, silently dropping its
+ # in-window records that lie beyond the limit. ``end_tx = None``
+ # (open-ended/current) means no upper bound for that kind.
+ bounds_by_kind: dict[str, tuple[int, int | None]] = {}
+ for (table_kind, _entity_id), windows in windows_by_entity.items():
+ for w in windows:
+ cur = bounds_by_kind.get(table_kind)
+ if cur is None:
+ bounds_by_kind[table_kind] = (w.start_tx, w.end_tx)
+ continue
+ lo = min(cur[0], w.start_tx)
+ hi = None if (cur[1] is None or w.end_tx is None) else max(cur[1], w.end_tx)
+ bounds_by_kind[table_kind] = (lo, hi)
+
+ rows, truncated, truncation_floor = _select_change_rows_for_kinds(
+ ids_by_kind, bounds_by_kind, since, until, _MAX_FETCHED_RECORDS
+ )
+ filtered = [
+ row
+ for row in rows
+ if row_within_any_window(
+ row, windows_by_entity.get((row["entity_kind"], row["entity_id"]), [])
+ )
+ ]
+ if truncated and truncation_floor is not None:
+ # Collapse truncation to a clean time cut. Below the highest per-kind
+ # fetch floor at least one kind is missing rows, so surfacing the
+ # other kinds there would present an incomplete ("nothing else
+ # changed") picture. Drop everything older than the floor so
+ # ``truncated=True`` means "complete at/after this instant, nothing
+ # shown before it" — a time boundary rather than per-kind holes.
+ filtered = [r for r in filtered if r["issued_at"] >= truncation_floor]
+ # Sort key must be TOTAL so pagination is stable across requests: two
+ # records from different entities can share (issued_at, transaction_id,
+ # sequence), so append (entity_kind, entity_id) to break remaining ties
+ # deterministically. Without these the relative order of tied records
+ # depends on set-iteration order and a record could shift pages.
+ filtered.sort(
+ key=lambda r: (
+ r["issued_at"],
+ r["transaction_id"],
+ r["sequence"],
+ r["entity_kind"],
+ r["entity_id"],
+ ),
+ reverse=True,
+ )
+ return filtered, truncated
+
+
+def _select_change_rows_for_kinds(
+ ids_by_kind: dict[str, set[int]],
+ bounds_by_kind: dict[str, tuple[int, int | None]],
+ since: datetime | None,
+ until: datetime | None,
+ limit: int,
+) -> tuple[list[dict[str, Any]], bool, datetime | None]:
+ """Fire one SELECT per entity_kind with ``entity_id IN (...)``;
+ concatenate the results. Each SELECT joins ``version_transaction``
+ + ``ab_user`` so the orchestrator has the columns it needs for
+ decoration.
+
+ Per-kind, not one query: SQLAlchemy's ``tuple_(entity_kind,
+ entity_id).in_(...)`` would collapse the three queries into one,
+ but its SQL emission is not portable across Postgres, MySQL, and
+ SQLite. The per-kind shape is the correct trade-off given
+ Superset's multi-dialect requirement (at most 3 round-trips per
+ request, bounded by the kind taxonomy). Do not "optimise" into a
+ composite-tuple IN clause without verifying the SQL on all three
+ dialects.
+
+ **Init-order dependency.** ``tx_tbl.c.action_kind`` resolves only
+ after ``init_versioning()`` has run — the column is appended onto
+ Continuum's transaction Table by
+ ``superset.versioning.factory.VersionTransactionFactory`` at app
+ start via ``append_column`` + ``add_property``. This helper is
+ safe to call from request-path code because the app is fully
+ initialised by then; calling it from a script that imports the
+ versioning package without going through ``init_versioning()``
+ will raise ``AttributeError`` on the ``action_kind`` attribute
+ access below."""
+ # pylint: disable=import-outside-toplevel
+ from sqlalchemy_continuum import versioning_manager
+
+ from superset import security_manager
+
+ tx_tbl = versioning_manager.transaction_cls.__table__
+ user_tbl = security_manager.user_model.__table__
+ vc = version_changes_table
+ join_tree = vc.join(tx_tbl, vc.c.transaction_id == tx_tbl.c.id).outerjoin(
+ user_tbl, tx_tbl.c.user_id == user_tbl.c.id
+ )
+ select_cols = (
+ vc.c.transaction_id,
+ vc.c.entity_kind,
+ vc.c.entity_id,
+ vc.c.sequence,
+ vc.c.kind,
+ vc.c.operation,
+ vc.c.path,
+ vc.c.from_value,
+ vc.c.to_value,
+ tx_tbl.c.issued_at,
+ tx_tbl.c.user_id,
+ # ``action_kind`` is the high-level avenue (restore / import /
+ # clone / NULL=ordinary save) stamped by the originating
+ # command via the change-record listener. All records sharing a
+ # ``transaction_id`` share the same value. The column is
+ # declared on the Continuum Table by ``VersionTransactionFactory``,
+ # so ``tx_tbl.c.action_kind`` resolves cleanly here. See
+ # the three change-record dimensions.
+ tx_tbl.c.action_kind,
+ user_tbl.c.id.label("changed_by_id"),
+ user_tbl.c.first_name,
+ user_tbl.c.last_name,
+ )
+
+ out: list[dict[str, Any]] = []
+ truncated = False
+ truncation_floors: list[datetime] = []
+ for table_kind, entity_ids in ids_by_kind.items():
+ # The fetch budget is per KIND, not per id-chunk. A kind whose
+ # entity_ids span several bind-variable chunks shares one ``limit``
+ # across those chunks, so the rows a single request materializes are
+ # bounded by ``n_kinds * limit`` (<= 3 * _MAX_FETCHED_RECORDS), not
+ # ``n_chunks * limit``. ``entity_ids`` is chunked to stay inside
+ # SQLite's ``SQLITE_MAX_VARIABLE_NUMBER`` floor (default 999 in many
+ # builds); Postgres + MySQL accept the full list but the chunk is
+ # dialect-agnostic for simplicity.
+ kind_rows: list[dict[str, Any]] = []
+ for chunk in chunked_ids(entity_ids):
+ remaining = limit - len(kind_rows)
+ if remaining <= 0:
+ break
+ stmt = (
+ sa.select(*select_cols)
+ .select_from(join_tree)
+ .where(
+ vc.c.entity_kind == table_kind,
+ vc.c.entity_id.in_(chunk),
+ )
+ )
+ # Bound by the kind's window union so the LIMIT picks in-window
+ # rows (see fetch_change_records). The per-entity window filter
+ # still runs in Python afterwards for exact membership.
+ tx_lo, tx_hi = bounds_by_kind[table_kind]
+ stmt = stmt.where(vc.c.transaction_id >= tx_lo)
+ if tx_hi is not None:
+ stmt = stmt.where(vc.c.transaction_id < tx_hi)
+ if since is not None:
+ stmt = stmt.where(tx_tbl.c.issued_at >= since)
+ if until is not None:
+ stmt = stmt.where(tx_tbl.c.issued_at < until)
+ # Match fetch_change_records' final Python sort key order
+ # (issued_at, transaction_id, sequence, entity_id) so the
+ # per-kind budget keeps exactly the rows the final sort ranks
+ # highest. entity_kind is constant within a per-kind statement.
+ stmt = stmt.order_by(
+ tx_tbl.c.issued_at.desc(),
+ vc.c.transaction_id.desc(),
+ vc.c.sequence.desc(),
+ vc.c.entity_id.desc(),
+ ).limit(remaining)
+ kind_rows.extend(
+ dict(row)
+ for row in db.session.connection().execute(stmt).mappings().all()
+ )
+ if len(kind_rows) >= limit:
+ # This kind exhausted its budget; older rows almost certainly
+ # exist beyond it. Record the oldest ``issued_at`` fetched for
+ # the kind so the caller can collapse truncation into a single
+ # clean time cut across all kinds (see fetch_change_records).
+ truncated = True
+ truncation_floors.append(min(r["issued_at"] for r in kind_rows))
+ out.extend(kind_rows)
+ truncation_floor = max(truncation_floors) if truncation_floors else None
+ return out, truncated, truncation_floor
+
+
+# Per-KIND safety ceiling on how many change rows a single activity request
+# will materialize for one entity kind (shared across that kind's id-chunks).
+# Bounds memory/CPU for a path entity with very long history or many related
+# entities to <= n_kinds * _MAX_FETCHED_RECORDS per request. When a kind
+# exhausts its budget the response is flagged ``truncated`` and the returned
+# stream is clamped to a clean time cut so clients see a complete window with
+# older records omitted, not per-kind holes.
+_MAX_FETCHED_RECORDS = 5000
Review Comment:
**Suggestion:** Add an explicit type annotation to this module-level constant to satisfy the type-hint requirement for relevant variables. [custom_rule]
**Severity Level:** Minor 🧹
<details>
<summary><b>Why it matters? ⭐ </b></summary>
This module-level constant is a relevant variable that can be annotated but currently has no type hint, so it violates the Python type-hint requirement.
</details>
<details>
<summary><b>Rule source 📖 </b></summary>
.cursor/rules/dev-standard.mdc (line 28)
</details>
[](https://app.codeant.ai/fix-in-ide?tool=cursor&prompt_id=f515c796c8424cb8a8eb4fd5e362fd2f&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset) [](https://app.codeant.ai/fix-in-ide?tool=vscode-claude&prompt_id=f515c796c8424cb8a8eb4fd5e362fd2f&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset)
*(Use Cmd/Ctrl + Click for best experience)*
<details>
<summary><b>Prompt for AI Agent 🤖 </b></summary>
```mdx
This is a comment left during a code review.
**Path:** superset/versioning/activity/queries.py
**Line:** 501:501
**Comment:**
*Custom Rule: Add an explicit type annotation to this module-level constant to satisfy the type-hint requirement for relevant variables.
Validate the correctness of the flagged issue. If correct, How can I resolve this? If you propose a fix, implement it and please make it concise.
Once fix is implemented, also check other comments on the same PR, and ask user if the user wants to fix the rest of the comments as well. if said yes, then fetch all the comments validate the correctness and implement a minimal fix
```
</details>
<a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F41076&comment_hash=9f89efbcddcf00af443e0af7b62d3039c23aad689319e71b70ff4ae1e386550a&reaction=like'>👍</a> | <a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F41076&comment_hash=9f89efbcddcf00af443e0af7b62d3039c23aad689319e71b70ff4ae1e386550a&reaction=dislike'>👎</a>
##########
superset/versioning/activity/queries.py:
##########
@@ -0,0 +1,754 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""DB-touching helpers for the activity-view read path.
+
+All Phase A relationship walks (``charts_attached_to_dashboard``,
+``datasets_used_by_chart``, ``batch_datasets_used_by_charts``),
+the Phase B change-record fetch (``fetch_change_records`` /
+``_select_change_rows_for_kinds``), the name-denormalization helpers
+(``_resolve_names_for_kind`` / ``apply_entity_name_denormalization``), the
+path-entity resolution helper (``resolve_path_entity``), and the
+tombstone-state lookup (``check_entity_tombstones``) live here.
+
+Each helper is a thin SELECT-and-shape function — no orchestration,
+no business logic. Callers in :mod:`scope`, :mod:`render`, and
+:mod:`orchestrator` compose them into the end-to-end request.
+
+**Inline imports.** Continuum's ``version_class`` / ``versioning_manager``
+and the Superset model classes are imported inside each helper because
+this package is loaded from ``init_versioning()`` before all SQLAlchemy
+mappers are configured.
+"""
+
+from __future__ import annotations
+
+import logging
+from datetime import datetime
+from typing import Any
+from uuid import UUID
+
+import sqlalchemy as sa
+from flask_appbuilder import Model
+
+from superset.extensions import db
+from superset.versioning.activity.kinds import (
+ API_KIND_TO_TABLE,
+ chunked_ids,
+ EntityWindows,
+ load_live_model,
+ NAME_COLUMN,
+ NOT_FOUND_EXC,
+ TABLE_KIND_TO_API,
+ Window,
+)
+from superset.versioning.activity.windows import row_within_any_window
+from superset.versioning.changes import version_changes_table
+
+logger = logging.getLogger(__name__)
+
+# ---- Path-entity resolution -----------------------------------------------
+
+
+def resolve_path_entity(model_cls: type[Model], entity_uuid: UUID) -> tuple[Any, int]:
+ """Resolve *entity_uuid* to ``(live_entity, entity_id)`` or raise a
+ typed 404.
+
+ Soft-delete handling is inherited transparently from
+ :func:`superset.versioning.queries.find_active_by_uuid` once it
+ learns to filter out ``deleted_at IS NOT NULL`` rows; at that point
+ soft-deleted paths will also raise here.
+ """
+ # pylint: disable=import-outside-toplevel
+ from superset.versioning.queries import find_active_by_uuid
+
+ entity = find_active_by_uuid(model_cls, entity_uuid)
+ if entity is None:
+ api_kind = model_cls.__name__
+ exc_cls = NOT_FOUND_EXC.get(api_kind)
+ if exc_cls is None:
+ raise LookupError(
+ f"Activity view does not support model class {api_kind!r}"
+ )
+ raise exc_cls(str(entity_uuid))
+ return entity, entity.id
+
+
+def first_tracked_tx(
+ model_cls: type[Model], entity_id: int, entity_uuid: UUID
+) -> int | None:
+ """Return the earliest Continuum transaction_id of the shadow rows
+ that belong to *this* live entity, matched on ``(id, uuid)``; ``None``
+ when the entity has no tracked history yet.
+
+ Used to lower-bound the self-scope window. Matching on the integer
+ ``id`` alone would inherit a previously hard-deleted entity's history
+ under id reuse (SQLite/MySQL reuse ``max(id)+1``); pinning the uuid
+ too — the same discrimination :func:`mark_first_tracked_saves` uses —
+ scopes the window to the current entity's own transactions.
+ """
+ # pylint: disable=import-outside-toplevel
+ from sqlalchemy_continuum import version_class
+
+ shadow_tbl = version_class(model_cls).__table__
+ stmt = sa.select(sa.func.min(shadow_tbl.c.transaction_id)).where(
+ shadow_tbl.c.id == entity_id,
+ shadow_tbl.c.uuid == entity_uuid,
+ )
+ return db.session.connection().execute(stmt).scalar()
+
+
+# ---- Phase A: relationship-traversal queries ------------------------------
+
+
+def charts_attached_to_dashboard(dashboard_id: int) -> list[tuple[int, Window]]:
+ """Return ``(slice_id, window)`` for every chart that has ever been on
+ *dashboard_id*, with each association's validity window in
+ transaction-id space.
+
+ Reads from ``dashboard_slices_version`` (Continuum's auto-generated
+ M2M shadow). Rows with ``operation_type = 2`` (DELETE) are excluded
+ so we don't synthesize a phantom window from a detachment row.
+ """
+ # pylint: disable=import-outside-toplevel
+ from sqlalchemy_continuum import version_class
+
+ from superset.models.dashboard import Dashboard
+
+ metadata = version_class(Dashboard).__table__.metadata
+ m2m_tbl = metadata.tables.get("dashboard_slices_version")
+ if m2m_tbl is None:
+ return []
+
+ rows = (
+ db.session.connection()
+ .execute(
+ sa.select(
+ m2m_tbl.c.slice_id,
+ m2m_tbl.c.transaction_id,
+ m2m_tbl.c.end_transaction_id,
+ ).where(
+ m2m_tbl.c.dashboard_id == dashboard_id,
+ m2m_tbl.c.operation_type != 2,
+ m2m_tbl.c.slice_id.is_not(None),
+ )
+ )
+ .all()
+ )
+ result: list[tuple[int, Window]] = []
+ for row in rows:
+ try:
+ window = Window(row[1], row[2])
+ except ValueError:
+ # A degenerate shadow row (end_tx <= start_tx) must not 500 the
+ # endpoint; skip it and leave a breadcrumb for investigation.
+ logger.warning(
+ "activity: skipping degenerate dashboard_slices_version row "
+ "(dashboard_id=%s, slice_id=%s, tx=%s, end_tx=%s)",
+ dashboard_id,
+ row[0],
+ row[1],
+ row[2],
+ )
+ continue
+ result.append((row[0], window))
+ return result
+
+
+def datasets_used_by_chart(slice_id: int) -> list[tuple[int, Window]]:
+ """Return ``(datasource_id, window)`` for every dataset that *slice_id*
+ has ever pointed at, with each association's validity window.
+
+ Single-slice form, used by ``_resolve_chart_scope`` where there
+ is only one chart to walk. The dashboard-scope path calls
+ :func:`batch_datasets_used_by_charts` instead so the query fires
+ once for all slices on the dashboard, not once per slice.
+
+ Reads from ``slices_version`` (the chart parent shadow). Filters to
+ ``datasource_type = 'table'`` because the activity view only follows
+ the chart → ``SqlaTable`` dependency edge (not legacy/other
+ datasources). Rows with ``operation_type = 2`` are excluded.
+ """
+ return batch_datasets_used_by_charts({slice_id}).get(slice_id, [])
+
+
+def batch_datasets_used_by_charts(
+ slice_ids: set[int],
+) -> dict[int, list[tuple[int, Window]]]:
+ """Batch form of :func:`datasets_used_by_chart`. Returns
+ ``{slice_id: [(dataset_id, window), ...]}`` in a single query so the
+ dashboard-scope walker doesn't fire one query per chart on the
+ dashboard. The previous per-slice shape became O(n_charts) round-
+ trips, which dominated ``get_activity`` latency on dashboards with
+ rich history (profile run 2026-05-26 showed `resolve_scope`
+ accounting for ~1.9s out of 4s p95).
+ """
+ if not slice_ids:
+ return {}
+
+ # pylint: disable=import-outside-toplevel
+ from sqlalchemy_continuum import version_class
+
+ from superset.models.slice import Slice
+
+ slices_tbl = version_class(Slice).__table__
+ grouped: dict[int, list[tuple[int, Window]]] = {}
+ # Chunk the IN-clause under SQLite's bind-variable floor (a dashboard can
+ # carry more charts than the floor allows in one statement).
+ rows: list[Any] = []
+ for chunk in chunked_ids(slice_ids):
+ rows.extend(
+ db.session.connection()
+ .execute(
+ sa.select(
+ slices_tbl.c.id,
+ slices_tbl.c.datasource_id,
+ slices_tbl.c.transaction_id,
+ slices_tbl.c.end_transaction_id,
+ ).where(
+ slices_tbl.c.id.in_(chunk),
+ slices_tbl.c.datasource_type == "table",
+ slices_tbl.c.operation_type != 2,
+ slices_tbl.c.datasource_id.is_not(None),
+ )
+ )
+ .mappings()
+ .all()
+ )
+ for row in rows:
+ try:
+ window = Window(row["transaction_id"], row["end_transaction_id"])
+ except ValueError:
+ # A degenerate shadow row (end_tx <= start_tx) must not 500 the
+ # endpoint; skip it and leave a breadcrumb for investigation.
+ logger.warning(
+ "activity: skipping degenerate slices_version row "
+ "(slice_id=%s, datasource_id=%s, tx=%s, end_tx=%s)",
+ row["id"],
+ row["datasource_id"],
+ row["transaction_id"],
+ row["end_transaction_id"],
+ )
+ continue
+ grouped.setdefault(row["id"], []).append((row["datasource_id"], window))
+ return grouped
+
+
+# ---- Phase B: change-record fetch -----------------------------------------
+
+
+def fetch_change_records(
+ entity_window_tuples: list[EntityWindows],
+ since: datetime | None,
+ until: datetime | None,
+) -> tuple[list[dict[str, Any]], bool]:
+ """Fetch all ``version_changes`` rows matching any of the supplied
+ entity-window tuples, joined with ``version_transaction`` for
+ ``issued_at`` and ``user_id``.
+
+ Each tuple is ``(api_kind, entity_id, [(start_tx, end_tx), ...])``;
+ a record matches when ``entity_kind`` equals the table-stored form
+ of *api_kind*, ``entity_id`` matches, and ``transaction_id`` falls
+ inside at least one of the entity's windows. ``since``/``until``
+ further restrict by ``issued_at``.
+
+ Implementation: one SELECT per kind with ``entity_id IN (...)`` and
+ a wide ``transaction_id`` bound (the union of all windows for that
+ kind). Per-window precision is applied in Python afterward. This
+ keeps the SQL shape proportional to the number of *kinds* (≤3) and
+ the bound proportional to the union of windows, not the cross-
+ product of (entity, window) — which previously generated one OR
+ clause per (entity, window) pair and hit SQLite's
+ ``SQLITE_MAX_EXPR_DEPTH`` limit on dashboards with many slices
+ or many historical attachment windows.
+
+ The visibility filter runs after this function (records
+ the requester can't read are silently dropped and must not
+ contribute to ``count``), so the orchestrator paginates in Python
+ over the filtered list — there is no DB-level page ``OFFSET`` here.
+ There IS a per-kind safety ceiling (``_MAX_FETCHED_RECORDS``) that
+ bounds how much history a single request materializes (to at most
+ ``n_kinds * _MAX_FETCHED_RECORDS``); when a kind exhausts it, the
+ second return value is ``True``, the returned stream is clamped to a
+ clean time cut (records older than the highest per-kind fetch floor
+ are dropped so truncation is a time boundary, not per-kind holes), and
+ the caller surfaces ``truncated`` on the response.
+
+ Returns ``(records, truncated)``. Records are ordered by
+ ``(issued_at DESC, transaction_id DESC, sequence DESC)`` — the
+ secondary keys break ties for the stable-ordering contract.
+ """
+ if not entity_window_tuples:
+ return [], False
+
+ # Group windows by (table_kind, entity_id) and by table_kind for SQL
+ # narrowing. The fetch is per-kind; the post-filter is per-entity.
+ windows_by_entity: dict[tuple[str, int], list[Window]] = {}
+ ids_by_kind: dict[str, set[int]] = {}
+ for api_kind, entity_id, windows in entity_window_tuples:
+ table_kind = API_KIND_TO_TABLE.get(api_kind)
+ if table_kind is None or not windows:
+ continue
+ ids_by_kind.setdefault(table_kind, set()).add(entity_id)
+ windows_by_entity.setdefault((table_kind, entity_id), []).extend(windows)
+
+ if not ids_by_kind:
+ return [], False
+
+ # Per-kind transaction_id bounds = the union of that kind's windows.
+ # Pushing these into the SQL WHERE ensures the per-statement LIMIT
+ # selects from IN-WINDOW rows. Without it, a related entity whose
+ # association window is far in the past would have the newest ``limit``
+ # (out-of-window) rows fetched and discarded, silently dropping its
+ # in-window records that lie beyond the limit. ``end_tx = None``
+ # (open-ended/current) means no upper bound for that kind.
+ bounds_by_kind: dict[str, tuple[int, int | None]] = {}
+ for (table_kind, _entity_id), windows in windows_by_entity.items():
+ for w in windows:
+ cur = bounds_by_kind.get(table_kind)
+ if cur is None:
+ bounds_by_kind[table_kind] = (w.start_tx, w.end_tx)
+ continue
+ lo = min(cur[0], w.start_tx)
+ hi = None if (cur[1] is None or w.end_tx is None) else max(cur[1], w.end_tx)
+ bounds_by_kind[table_kind] = (lo, hi)
+
+ rows, truncated, truncation_floor = _select_change_rows_for_kinds(
+ ids_by_kind, bounds_by_kind, since, until, _MAX_FETCHED_RECORDS
+ )
+ filtered = [
+ row
+ for row in rows
+ if row_within_any_window(
+ row, windows_by_entity.get((row["entity_kind"], row["entity_id"]), [])
+ )
+ ]
+ if truncated and truncation_floor is not None:
+ # Collapse truncation to a clean time cut. Below the highest per-kind
+ # fetch floor at least one kind is missing rows, so surfacing the
+ # other kinds there would present an incomplete ("nothing else
+ # changed") picture. Drop everything older than the floor so
+ # ``truncated=True`` means "complete at/after this instant, nothing
+ # shown before it" — a time boundary rather than per-kind holes.
+ filtered = [r for r in filtered if r["issued_at"] >= truncation_floor]
+ # Sort key must be TOTAL so pagination is stable across requests: two
+ # records from different entities can share (issued_at, transaction_id,
+ # sequence), so append (entity_kind, entity_id) to break remaining ties
+ # deterministically. Without these the relative order of tied records
+ # depends on set-iteration order and a record could shift pages.
+ filtered.sort(
+ key=lambda r: (
+ r["issued_at"],
+ r["transaction_id"],
+ r["sequence"],
+ r["entity_kind"],
+ r["entity_id"],
+ ),
+ reverse=True,
+ )
+ return filtered, truncated
+
+
+def _select_change_rows_for_kinds(
+ ids_by_kind: dict[str, set[int]],
+ bounds_by_kind: dict[str, tuple[int, int | None]],
+ since: datetime | None,
+ until: datetime | None,
+ limit: int,
+) -> tuple[list[dict[str, Any]], bool, datetime | None]:
+ """Fire one SELECT per entity_kind with ``entity_id IN (...)``;
+ concatenate the results. Each SELECT joins ``version_transaction``
+ + ``ab_user`` so the orchestrator has the columns it needs for
+ decoration.
+
+ Per-kind, not one query: SQLAlchemy's ``tuple_(entity_kind,
+ entity_id).in_(...)`` would collapse the three queries into one,
+ but its SQL emission is not portable across Postgres, MySQL, and
+ SQLite. The per-kind shape is the correct trade-off given
+ Superset's multi-dialect requirement (at most 3 round-trips per
+ request, bounded by the kind taxonomy). Do not "optimise" into a
+ composite-tuple IN clause without verifying the SQL on all three
+ dialects.
+
+ **Init-order dependency.** ``tx_tbl.c.action_kind`` resolves only
+ after ``init_versioning()`` has run — the column is appended onto
+ Continuum's transaction Table by
+ ``superset.versioning.factory.VersionTransactionFactory`` at app
+ start via ``append_column`` + ``add_property``. This helper is
+ safe to call from request-path code because the app is fully
+ initialised by then; calling it from a script that imports the
+ versioning package without going through ``init_versioning()``
+ will raise ``AttributeError`` on the ``action_kind`` attribute
+ access below."""
+ # pylint: disable=import-outside-toplevel
+ from sqlalchemy_continuum import versioning_manager
+
+ from superset import security_manager
+
+ tx_tbl = versioning_manager.transaction_cls.__table__
+ user_tbl = security_manager.user_model.__table__
+ vc = version_changes_table
+ join_tree = vc.join(tx_tbl, vc.c.transaction_id == tx_tbl.c.id).outerjoin(
+ user_tbl, tx_tbl.c.user_id == user_tbl.c.id
+ )
+ select_cols = (
+ vc.c.transaction_id,
+ vc.c.entity_kind,
+ vc.c.entity_id,
+ vc.c.sequence,
+ vc.c.kind,
+ vc.c.operation,
+ vc.c.path,
+ vc.c.from_value,
+ vc.c.to_value,
+ tx_tbl.c.issued_at,
+ tx_tbl.c.user_id,
+ # ``action_kind`` is the high-level avenue (restore / import /
+ # clone / NULL=ordinary save) stamped by the originating
+ # command via the change-record listener. All records sharing a
+ # ``transaction_id`` share the same value. The column is
+ # declared on the Continuum Table by ``VersionTransactionFactory``,
+ # so ``tx_tbl.c.action_kind`` resolves cleanly here. See
+ # the three change-record dimensions.
+ tx_tbl.c.action_kind,
+ user_tbl.c.id.label("changed_by_id"),
+ user_tbl.c.first_name,
+ user_tbl.c.last_name,
+ )
+
+ out: list[dict[str, Any]] = []
+ truncated = False
+ truncation_floors: list[datetime] = []
+ for table_kind, entity_ids in ids_by_kind.items():
+ # The fetch budget is per KIND, not per id-chunk. A kind whose
+ # entity_ids span several bind-variable chunks shares one ``limit``
+ # across those chunks, so the rows a single request materializes are
+ # bounded by ``n_kinds * limit`` (<= 3 * _MAX_FETCHED_RECORDS), not
+ # ``n_chunks * limit``. ``entity_ids`` is chunked to stay inside
+ # SQLite's ``SQLITE_MAX_VARIABLE_NUMBER`` floor (default 999 in many
+ # builds); Postgres + MySQL accept the full list but the chunk is
+ # dialect-agnostic for simplicity.
+ kind_rows: list[dict[str, Any]] = []
+ for chunk in chunked_ids(entity_ids):
+ remaining = limit - len(kind_rows)
+ if remaining <= 0:
+ break
+ stmt = (
+ sa.select(*select_cols)
+ .select_from(join_tree)
+ .where(
+ vc.c.entity_kind == table_kind,
+ vc.c.entity_id.in_(chunk),
+ )
+ )
+ # Bound by the kind's window union so the LIMIT picks in-window
+ # rows (see fetch_change_records). The per-entity window filter
+ # still runs in Python afterwards for exact membership.
+ tx_lo, tx_hi = bounds_by_kind[table_kind]
+ stmt = stmt.where(vc.c.transaction_id >= tx_lo)
+ if tx_hi is not None:
+ stmt = stmt.where(vc.c.transaction_id < tx_hi)
+ if since is not None:
+ stmt = stmt.where(tx_tbl.c.issued_at >= since)
+ if until is not None:
+ stmt = stmt.where(tx_tbl.c.issued_at < until)
+ # Match fetch_change_records' final Python sort key order
+ # (issued_at, transaction_id, sequence, entity_id) so the
+ # per-kind budget keeps exactly the rows the final sort ranks
+ # highest. entity_kind is constant within a per-kind statement.
+ stmt = stmt.order_by(
+ tx_tbl.c.issued_at.desc(),
+ vc.c.transaction_id.desc(),
+ vc.c.sequence.desc(),
+ vc.c.entity_id.desc(),
+ ).limit(remaining)
+ kind_rows.extend(
+ dict(row)
+ for row in db.session.connection().execute(stmt).mappings().all()
+ )
+ if len(kind_rows) >= limit:
+ # This kind exhausted its budget; older rows almost certainly
+ # exist beyond it. Record the oldest ``issued_at`` fetched for
+ # the kind so the caller can collapse truncation into a single
+ # clean time cut across all kinds (see fetch_change_records).
+ truncated = True
+ truncation_floors.append(min(r["issued_at"] for r in kind_rows))
+ out.extend(kind_rows)
+ truncation_floor = max(truncation_floors) if truncation_floors else None
+ return out, truncated, truncation_floor
+
+
+# Per-KIND safety ceiling on how many change rows a single activity request
+# will materialize for one entity kind (shared across that kind's id-chunks).
+# Bounds memory/CPU for a path entity with very long history or many related
+# entities to <= n_kinds * _MAX_FETCHED_RECORDS per request. When a kind
+# exhausts its budget the response is flagged ``truncated`` and the returned
+# stream is clamped to a clean time cut so clients see a complete window with
+# older records omitted, not per-kind holes.
+_MAX_FETCHED_RECORDS = 5000
+
+
+def mark_first_tracked_saves(records: list[dict[str, Any]]) -> None:
+ """Set ``first_tracked_save`` on each record in place: ``True`` when
+ the record's transaction is the entity's FIRST UPDATE (op=1) in its
+ shadow table.
+
+ The first save of an entity that predates versioning replays every
+ params-normalization delta against the retroactive baseline — a
+ legacy chart's first Explore save produced ~74 records in one
+ transaction (version-history UI feedback). The server
+ can't distinguish "normalization" from "the user changed 74 things",
+ but it CAN say "this was the entity's first tracked save"; clients
+ use the marker to collapse such transactions.
+
+ One ``GROUP BY`` query per kind (≤3), chunked like the record fetch.
+ Shadow rows are matched on ``(id, uuid)`` against the live row — a
+ bare ``id`` match would inherit a previously hard-deleted entity's
+ history under id reuse (SQLite/MySQL reuse ``max(id)+1``) and mark
+ the wrong transaction. Consequence: hard-deleted entities (no live
+ row) and NULL-uuid shadow rows never get a ``True`` marker — their
+ records always carry ``first_tracked_save=False``. Mutates *records*
+ in place — same contract as the other decoration passes in
+ :mod:`superset.versioning.activity.render`.
+ """
+ if not records:
+ return
+ # pylint: disable=import-outside-toplevel
+ from sqlalchemy_continuum import version_class
+
+ ids_by_kind: dict[str, set[int]] = {}
+ for r in records:
+ ids_by_kind.setdefault(r["entity_kind"], set()).add(r["entity_id"])
+
+ first_tx_by_entity: dict[tuple[str, int], int] = {}
+ for table_kind, entity_ids in ids_by_kind.items():
+ model_name = TABLE_KIND_TO_API.get(table_kind)
+ if model_name is None:
+ continue
+ live_model = load_live_model(model_name)
+ live_tbl = live_model.__table__
+ shadow_tbl = version_class(live_model).__table__
+ for chunk in chunked_ids(entity_ids):
+ stmt = (
+ sa.select(
+ shadow_tbl.c.id,
+ sa.func.min(shadow_tbl.c.transaction_id),
+ )
+ .select_from(
+ shadow_tbl.join(
+ live_tbl,
+ sa.and_(
+ shadow_tbl.c.id == live_tbl.c.id,
+ shadow_tbl.c.uuid == live_tbl.c.uuid,
+ ),
+ )
+ )
+ .where(
+ shadow_tbl.c.operation_type == 1,
+ shadow_tbl.c.id.in_(chunk),
+ )
+ .group_by(shadow_tbl.c.id)
+ )
+ for entity_id, min_tx in db.session.connection().execute(stmt):
+ first_tx_by_entity[(table_kind, entity_id)] = min_tx
+
+ for r in records:
+ r["first_tracked_save"] = (
+ first_tx_by_entity.get((r["entity_kind"], r["entity_id"]))
+ == r["transaction_id"]
+ )
+
+
+# ---- Name denormalization -------------------------------------------------
+
+
+def _resolve_names_for_kind(
+ api_kind: str, pairs: set[tuple[int, int]]
+) -> dict[tuple[int, int], str]:
+ """For one entity kind, return ``{(entity_id, target_tx): name}`` from
+ the shadow row valid at *target_tx* (validity-strategy predicate).
+ Empty mapping when the kind has no name column registered.
+ """
+ # pylint: disable=import-outside-toplevel
+ from sqlalchemy_continuum import version_class
+
+ if api_kind not in NAME_COLUMN or not pairs:
+ return {}
+
+ model_name, name_col = NAME_COLUMN[api_kind]
+ model_cls = load_live_model(model_name)
+ shadow_tbl = version_class(model_cls).__table__
+ ids = {eid for eid, _ in pairs}
+ # Bound the scan to the transaction range the page-set actually needs.
+ # A row valid at some target_tx must start at/before the newest target
+ # and stay open past the oldest one, so rows outside
+ # ``[min_target_tx, max_target_tx]`` can never win the validity match.
+ # Without this bound the query loads every historical version row of
+ # every referenced entity — an entity with thousands of versions would
+ # defeat the memory ceiling the change-record fetch enforces.
+ target_txs = {target_tx for _, target_tx in pairs}
+ min_tx, max_tx = min(target_txs), max(target_txs)
+ per_entity: dict[int, list[tuple[int, int | None, Any]]] = {}
+ # Chunk the IN-clause to stay under SQLite's bind-variable floor (the
+ # same reason _select_change_rows_for_kinds chunks).
+ for chunk in chunked_ids(ids):
+ rows = (
+ db.session.connection()
+ .execute(
+ sa.select(
+ shadow_tbl.c.id,
+ shadow_tbl.c.transaction_id,
+ shadow_tbl.c.end_transaction_id,
+ shadow_tbl.c[name_col],
+ ).where(
+ shadow_tbl.c.id.in_(chunk),
+ shadow_tbl.c.transaction_id <= max_tx,
+ sa.or_(
+ shadow_tbl.c.end_transaction_id.is_(None),
+ shadow_tbl.c.end_transaction_id > min_tx,
+ ),
+ )
+ )
+ .all()
+ )
+ for row in rows:
+ per_entity.setdefault(row[0], []).append((row[1], row[2], row[3]))
+
+ resolved: dict[tuple[int, int], str] = {}
+ for entity_id, target_tx in pairs:
+ for start_tx, end_tx, name in per_entity.get(entity_id, []):
+ if start_tx <= target_tx and (end_tx is None or end_tx > target_tx):
+ # Coerce a NULL name (e.g. a DELETE shadow row winning the
+ # validity match) to "" so entity_name is never None.
+ resolved[(entity_id, target_tx)] = name if name is not None else ""
+ break
+ return resolved
+
+
+def apply_entity_name_denormalization(records: list[dict[str, Any]]) -> None:
+ """Resolve each record's ``entity_name`` from the shadow row valid at
+ its ``transaction_id``. Adds an ``entity_name`` key to every record
+ in place; returns ``None``.
+
+ The lookup is per (table-stored ``entity_kind``, ``entity_id``,
+ ``transaction_id``) triple. One ``IN``-clause query per kind keeps
+ round-trips bounded by the number of distinct kinds (≤3) regardless
+ of result-set size. The in-place mutation avoids re-allocating
+ thousands of dicts on hot dashboards; the name + return signature
+ make the side effect explicit instead of pretending to be a pure
+ projection.
+ """
+ if not records:
+ return
+
+ needed_by_kind: dict[str, set[tuple[int, int]]] = {}
+ for record in records:
+ api_kind = TABLE_KIND_TO_API.get(record["entity_kind"])
+ if api_kind is None or api_kind not in NAME_COLUMN:
+ continue
+ needed_by_kind.setdefault(api_kind, set()).add(
+ (record["entity_id"], record["transaction_id"])
+ )
+
+ resolved: dict[tuple[str, int, int], str] = {}
+ for api_kind, pairs in needed_by_kind.items():
+ for (entity_id, target_tx), name in _resolve_names_for_kind(
+ api_kind, pairs
+ ).items():
+ resolved[(api_kind, entity_id, target_tx)] = name
+
+ for record in records:
+ api_kind_for_record = TABLE_KIND_TO_API.get(record["entity_kind"], "")
+ key = (api_kind_for_record, record["entity_id"], record["transaction_id"])
+ record["entity_name"] = resolved.get(key, "")
+
+
+# ---- Live-row existence + soft-delete state -------------------------------
+
+
+def check_entity_tombstones(
+ distinct_entities: set[tuple[str, int]],
+) -> dict[tuple[str, int], dict[str, Any]]:
+ """For each ``(api_kind, entity_id)``, report ``deleted`` (no live
+ row) and ``deletion_state`` (``"soft_deleted"`` iff the live row has
+ a non-null ``deleted_at``, else ``None``).
+
+ The ``deleted_at`` column is probed for at runtime: when the model
+ classes don't have one, entities are reported as never soft-deleted
+ (``deletion_state=None``); when a ``deleted_at`` column exists, this
+ helper picks it up automatically.
+ """
+ result: dict[tuple[str, int], dict[str, Any]] = {}
+ if not distinct_entities:
+ return result
+
+ by_kind: dict[str, list[int]] = {}
+ for api_kind, entity_id in distinct_entities:
+ by_kind.setdefault(api_kind, []).append(entity_id)
+
+ # ``no_autoflush`` mirrors the defensive posture of the listener-
+ # side reads. Today's callers run from request-path code with no
+ # pending writes; a future caller that probes tombstones before a
+ # flush would otherwise trigger autoflush mid-read.
+ with db.session.no_autoflush:
+ for api_kind, entity_ids in by_kind.items():
+ for entity_id, state in _tombstone_states_for_kind(
+ api_kind, entity_ids
+ ).items():
+ result[(api_kind, entity_id)] = state
+ return result
+
+
+_TOMBSTONE = {"deleted": True, "deletion_state": None}
Review Comment:
**Suggestion:** Add an explicit type annotation to this module-level dictionary constant so its structure is enforced by type checking. [custom_rule]
**Severity Level:** Minor 🧹
<details>
<summary><b>Why it matters? ⭐ </b></summary>
This module-level dictionary constant is also a relevant variable that can be annotated, but it is left untyped, so the type-hint rule is violated.
</details>
<details>
<summary><b>Rule source 📖 </b></summary>
.cursor/rules/dev-standard.mdc (line 28)
</details>
[](https://app.codeant.ai/fix-in-ide?tool=cursor&prompt_id=a472c9c07fbe4fce99123c577d912ea1&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset) [](https://app.codeant.ai/fix-in-ide?tool=vscode-claude&prompt_id=a472c9c07fbe4fce99123c577d912ea1&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset)
*(Use Cmd/Ctrl + Click for best experience)*
<details>
<summary><b>Prompt for AI Agent 🤖 </b></summary>
```mdx
This is a comment left during a code review.
**Path:** superset/versioning/activity/queries.py
**Line:** 715:715
**Comment:**
*Custom Rule: Add an explicit type annotation to this module-level dictionary constant so its structure is enforced by type checking.
Validate the correctness of the flagged issue. If correct, How can I resolve this? If you propose a fix, implement it and please make it concise.
Once fix is implemented, also check other comments on the same PR, and ask user if the user wants to fix the rest of the comments as well. if said yes, then fetch all the comments validate the correctness and implement a minimal fix
```
</details>
<a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F41076&comment_hash=629e433c5632d2e716c321c94378478d5aa5ae4b65b6fe3a663b3246738b99de&reaction=like'>👍</a> | <a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F41076&comment_hash=629e433c5632d2e716c321c94378478d5aa5ae4b65b6fe3a663b3246738b99de&reaction=dislike'>👎</a>
##########
superset/versioning/activity/scope.py:
##########
@@ -0,0 +1,118 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Scope resolution — turn a path entity into the related-entity walk.
+
+Composes :mod:`~superset.versioning.activity.queries` (Phase A
+relationship walks) and :mod:`~superset.versioning.activity.windows`
+(pure interval arithmetic) into the
+``list[EntityWindows]`` scope that
+:func:`~superset.versioning.activity.queries.fetch_change_records`
+consumes.
+
+The functions here read the DB (via the Phase A helpers in
+:mod:`~superset.versioning.activity.queries`); the pure window-
+arithmetic functions previously colocated here now live in
+:mod:`~superset.versioning.activity.windows` so the package no longer
+needs a lazy import to dodge a ``scope ↔ queries`` cycle.
+"""
+
+from __future__ import annotations
+
+from superset.versioning.activity.kinds import EntityWindows, Window
+from superset.versioning.activity.queries import (
+ batch_datasets_used_by_charts,
+ charts_attached_to_dashboard,
+ datasets_used_by_chart,
+)
+from superset.versioning.activity.windows import (
+ intersect_windows,
+ merge_entity_windows,
+)
+
+
+def resolve_scope(
+ path_kind: str,
+ path_id: int,
+ include: str,
+ self_start_tx: int | None = None,
+) -> list[EntityWindows]:
+ """Build the ``[(api_kind, entity_id, [windows])]`` list that
+ :func:`~superset.versioning.activity.queries.fetch_change_records`
+ consumes, branching by *path_kind* and *include* mode.
+
+ *self_start_tx* lower-bounds the self window at the path entity's first
+ tracked transaction (see
+ :func:`~superset.versioning.activity.queries.first_tracked_tx`). This
+ scopes the self stream to the current entity's own history: matching
+ self records on the bare integer id would otherwise inherit a
+ hard-deleted predecessor's change records under id reuse
+ (SQLite/MySQL reuse ``max(id)+1``). ``None`` means the entity has no
+ tracked history yet, so the self stream contributes nothing.
+ """
+ want_self = include in ("all", "self")
+ want_related = include in ("all", "related")
Review Comment:
**Suggestion:** Add an explicit boolean type annotation to this local control variable to comply with the type-hint policy. [custom_rule]
**Severity Level:** Minor 🧹
<details>
<summary><b>Why it matters? ⭐ </b></summary>
This is another newly introduced local boolean control variable that can be annotated but is not. That fits the type-hint requirement for relevant variables.
</details>
<details>
<summary><b>Rule source 📖 </b></summary>
.cursor/rules/dev-standard.mdc (line 28)
</details>
[](https://app.codeant.ai/fix-in-ide?tool=cursor&prompt_id=dd75b833bdf346e7b7545b3971e4cc29&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset) [](https://app.codeant.ai/fix-in-ide?tool=vscode-claude&prompt_id=dd75b833bdf346e7b7545b3971e4cc29&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset)
*(Use Cmd/Ctrl + Click for best experience)*
<details>
<summary><b>Prompt for AI Agent 🤖 </b></summary>
```mdx
This is a comment left during a code review.
**Path:** superset/versioning/activity/scope.py
**Line:** 67:67
**Comment:**
*Custom Rule: Add an explicit boolean type annotation to this local control variable to comply with the type-hint policy.
Validate the correctness of the flagged issue. If correct, How can I resolve this? If you propose a fix, implement it and please make it concise.
Once fix is implemented, also check other comments on the same PR, and ask user if the user wants to fix the rest of the comments as well. if said yes, then fetch all the comments validate the correctness and implement a minimal fix
```
</details>
<a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F41076&comment_hash=11e2a9768cb4652c6871b437ce4dcfbdf503c3eab1ba71b8921692eb8b28f903&reaction=like'>👍</a> | <a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F41076&comment_hash=11e2a9768cb4652c6871b437ce4dcfbdf503c3eab1ba71b8921692eb8b28f903&reaction=dislike'>👎</a>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(versioning): cross-entity version activity view [superset]
Posted by "codeant-ai-for-open-source[bot] (via GitHub)" <gi...@apache.org>.
codeant-ai-for-open-source[bot] commented on code in PR #41076:
URL: https://github.com/apache/superset/pull/41076#discussion_r3560195187
##########
superset/versioning/activity/windows.py:
##########
@@ -0,0 +1,108 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Pure window arithmetic on half-open ``[start_tx, end_tx)`` intervals.
+
+Extracted from the DB-touching scope resolution so that:
+
+* :mod:`scope` (DB-touching) can import this module at module-top.
+* :mod:`queries.fetch_change_records` can import
+ :func:`row_within_any_window` at module-top instead of through a
+ lazy import that previously dodged a ``scope ↔ queries`` cycle.
+
+Everything here is pure Python — no DB, no Flask. ``end_tx = None``
+means "open-ended (current)" and behaves like positive infinity.
+"""
+
+from __future__ import annotations
+
+from typing import Any
+
+from superset.versioning.activity.kinds import EntityWindows, Window
+
+
+def intersect_windows(outer: Window, inner: Window) -> Window | None:
+ """Intersect two half-open ``[start_tx, end_tx)`` windows.
+
+ Returns the clipped overlap, or ``None`` when they are disjoint.
+ ``end_tx = None`` means "open ended (current)" and acts like
+ positive infinity. Thin wrapper over :meth:`Window.intersect` —
+ kept as a free function so callers and tests don't have to migrate
+ to method form in lockstep with the dataclass promotion.
+ """
+ return outer.intersect(inner)
+
+
+def row_within_any_window(row: dict[str, Any], windows: list[Window]) -> bool:
+ """``True`` iff ``row['transaction_id']`` falls inside at least one
+ of *windows*. Half-open interval semantics match
+ :func:`intersect_windows`."""
+ if not windows:
+ return False
+ tx_id = row["transaction_id"]
+ return any(w.contains(tx_id) for w in windows)
+
+
+def merge_entity_windows(scope: list[EntityWindows]) -> list[EntityWindows]:
+ """Collapse repeated ``(api_kind, entity_id)`` entries by unioning
+ their window lists, and collapse overlapping/touching windows
+ within each entity into one.
+
+ The OR-clause in
+ :func:`~superset.versioning.activity.queries.fetch_change_records`
+ generates one branch per (kind, id, window) tuple. Without the
+ within-entity union, a chart that's been attached-and-detached
+ many times (or that repeated fixture loads have populated the M2M
+ shadow for) yields a separate clause per redundant window — at
+ ~10 entities × ~50 windows the SQL hits SQLite's
+ ``SQLITE_MAX_EXPR_DEPTH`` (1000). Merging here keeps the clause
+ count proportional to the number of *distinct* validity intervals,
+ not the number of shadow rows.
+ """
+ merged: dict[tuple[str, int], list[Window]] = {}
+ for api_kind, entity_id, windows in scope:
+ merged.setdefault((api_kind, entity_id), []).extend(windows)
+ return [
+ (api_kind, entity_id, union_windows(windows))
+ for (api_kind, entity_id), windows in merged.items()
+ ]
+
+
+def union_windows(windows: list[Window]) -> list[Window]:
+ """Sort + merge overlapping/touching half-open intervals.
+
+ Pure function — no DB. Touching ``[a, b)`` and ``[b, c)`` merge into
+ ``[a, c)``. ``end_tx = None`` (open-ended) absorbs everything to its
+ right. Returns a minimal disjoint cover of the input set.
+ """
+ if not windows:
+ return []
+ sorted_windows = sorted(windows, key=lambda w: w.start_tx)
Review Comment:
**Suggestion:** Add a concrete type annotation for this derived windows collection so the new function keeps explicit typing for key intermediate variables. [custom_rule]
**Severity Level:** Minor 🧹
<details>
<summary><b>Why it matters? ⭐ </b></summary>
This intermediate collection is a relevant local variable in newly added Python code and can be explicitly annotated as list[Window]. The custom rule flags new or modified Python code that omits type hints on relevant variables that can be annotated, so the suggestion matches a real violation.
</details>
<details>
<summary><b>Rule source 📖 </b></summary>
.cursor/rules/dev-standard.mdc (line 28)
</details>
[](https://app.codeant.ai/fix-in-ide?tool=cursor&prompt_id=09a904a333c74b93972f997b2863f50b&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset) [](https://app.codeant.ai/fix-in-ide?tool=vscode-claude&prompt_id=09a904a333c74b93972f997b2863f50b&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset)
*(Use Cmd/Ctrl + Click for best experience)*
<details>
<summary><b>Prompt for AI Agent 🤖 </b></summary>
```mdx
This is a comment left during a code review.
**Path:** superset/versioning/activity/windows.py
**Line:** 93:93
**Comment:**
*Custom Rule: Add a concrete type annotation for this derived windows collection so the new function keeps explicit typing for key intermediate variables.
Validate the correctness of the flagged issue. If correct, How can I resolve this? If you propose a fix, implement it and please make it concise.
Once fix is implemented, also check other comments on the same PR, and ask user if the user wants to fix the rest of the comments as well. if said yes, then fetch all the comments validate the correctness and implement a minimal fix
```
</details>
<a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F41076&comment_hash=fd1a252863ba262f0e948deed0a327cbffed0a3042b144b242c2648b3dbdb04b&reaction=like'>👍</a> | <a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F41076&comment_hash=fd1a252863ba262f0e948deed0a327cbffed0a3042b144b242c2648b3dbdb04b&reaction=dislike'>👎</a>
##########
superset/versioning/activity/orchestrator.py:
##########
@@ -0,0 +1,434 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Top-level orchestrator + query-param parsing + observability.
+
+This is the public entry point for the activity-view read path. One
+function — :func:`get_activity` — dispatches on the path entity's
+model class to assemble the cross-entity activity stream:
+
+1. ``resolve_path_entity`` (queries.py) — resolve UUID → live entity.
+2. ``resolve_scope`` (scope.py) — build the related-entity window list.
+3. ``fetch_change_records`` (queries.py) — pull rows from
+ ``version_changes`` joined with ``version_transaction`` and ``ab_user``.
+4. ``filter_records_by_visibility`` (visibility.py) — silent
+ drop of records the requester can't read.
+5. ``apply_entity_name_denormalization`` (queries.py) — resolve entity names
+ from the shadow row valid at each record's transaction_id.
+6. ``apply_record_decoration`` (render.py) — synthesize the ActivityRecord
+ DTO fields and strip internal-only columns.
+7. Paginate in Python over the post-filter list.
+
+Parameter parsing for the REST endpoints lives here too —
+:func:`parse_activity_query_params` is called by the three
+``/activity/`` endpoint handlers before they call ``get_activity``.
+Same for the observability instrumentation: ``_phase_timer`` and
+``_emit_request_shape_attributes`` emit the per-phase timing and
+request-shape metrics, on the same prefix the cross-coupling test pins.
+"""
+
+from __future__ import annotations
+
+import contextlib
+from collections.abc import Iterator
+from datetime import datetime, timezone
+from typing import Any
+from uuid import UUID
+
+from flask import Response
+from flask_appbuilder import Model
+
+from superset.utils import json
+from superset.versioning.activity.kinds import EntityWindows
+from superset.versioning.activity.queries import (
+ apply_entity_name_denormalization,
+ fetch_change_records,
+ first_tracked_tx,
+ mark_first_tracked_saves,
+ resolve_path_entity,
+)
+from superset.versioning.activity.render import apply_record_decoration
+from superset.versioning.activity.scope import resolve_scope
+from superset.versioning.activity.visibility import filter_records_by_visibility
+from superset.versioning.api_helpers import (
+ PathEntityResponseError,
+ resolve_endpoint_path_entity,
+)
+
+_DEFAULT_PAGE_SIZE = 25
+_MAX_PAGE_SIZE = 200
+_VALID_INCLUDE_VALUES: frozenset[str] = frozenset({"self", "related", "all"})
+
+
+# Upper bound on the ``q`` search string. The search is a substring scan over
+# the (already-capped) materialized record set, so this is a cheap-DoS guard,
+# not a correctness limit.
+_MAX_Q_LENGTH = 1024
Review Comment:
**Suggestion:** Add explicit type annotations to these new module-level constants to satisfy the strict type-hinting rule for relevant variables. [custom_rule]
**Severity Level:** Minor 🧹
<details>
<summary><b>Why it matters? ⭐ </b></summary>
These new module-level constants are plain assignments without type annotations, and they are simple scalar values that can be annotated. This matches the rule requiring type hints for relevant new or modified Python variables.
</details>
<details>
<summary><b>Rule source 📖 </b></summary>
.cursor/rules/dev-standard.mdc (line 28)
</details>
[](https://app.codeant.ai/fix-in-ide?tool=cursor&prompt_id=287864ef3be7455aa1c6c20addbada31&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset) [](https://app.codeant.ai/fix-in-ide?tool=vscode-claude&prompt_id=287864ef3be7455aa1c6c20addbada31&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset)
*(Use Cmd/Ctrl + Click for best experience)*
<details>
<summary><b>Prompt for AI Agent 🤖 </b></summary>
```mdx
This is a comment left during a code review.
**Path:** superset/versioning/activity/orchestrator.py
**Line:** 71:79
**Comment:**
*Custom Rule: Add explicit type annotations to these new module-level constants to satisfy the strict type-hinting rule for relevant variables.
Validate the correctness of the flagged issue. If correct, How can I resolve this? If you propose a fix, implement it and please make it concise.
Once fix is implemented, also check other comments on the same PR, and ask user if the user wants to fix the rest of the comments as well. if said yes, then fetch all the comments validate the correctness and implement a minimal fix
```
</details>
<a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F41076&comment_hash=0aa50aaad7536d989b39e89df1dc66c7e0c970d3b38cce27131041420321cd05&reaction=like'>👍</a> | <a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F41076&comment_hash=0aa50aaad7536d989b39e89df1dc66c7e0c970d3b38cce27131041420321cd05&reaction=dislike'>👎</a>
##########
superset/versioning/changes/listener.py:
##########
@@ -198,6 +198,29 @@ def build_action_headline(
# is correctly deduped.
_REGISTERED_SENTINEL = "_versioning_change_listener_registered"
+#: Metric namespace for swallowed capture-path failures. The capture
+#: listeners fail open (a versioning bug must never break a user's save),
+#: so the read path (``activity/orchestrator``) is richly instrumented but
+#: the write path historically logged-and-swallowed with no counter. Each
+#: ``_incr_capture_error(stage)`` emits ``<prefix>.<stage>.error`` so a
+#: systematic capture regression is alertable rather than log-grep-only.
+_CAPTURE_METRIC_PREFIX = "superset.versioning.capture"
Review Comment:
**Suggestion:** Add an explicit type annotation to this new module-level constant to satisfy the type-hint requirement for annotatable variables. [custom_rule]
**Severity Level:** Minor 🧹
<details>
<summary><b>Why it matters? ⭐ </b></summary>
This is a new module-level variable in Python and it is annotatable, but the code omits a type hint. That matches the llmcfg-require-python-type-hints rule.
</details>
<details>
<summary><b>Rule source 📖 </b></summary>
.cursor/rules/dev-standard.mdc (line 28)
</details>
[](https://app.codeant.ai/fix-in-ide?tool=cursor&prompt_id=aae8c1f98f8f40d9af2fd564a0dd8881&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset) [](https://app.codeant.ai/fix-in-ide?tool=vscode-claude&prompt_id=aae8c1f98f8f40d9af2fd564a0dd8881&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset)
*(Use Cmd/Ctrl + Click for best experience)*
<details>
<summary><b>Prompt for AI Agent 🤖 </b></summary>
```mdx
This is a comment left during a code review.
**Path:** superset/versioning/changes/listener.py
**Line:** 207:207
**Comment:**
*Custom Rule: Add an explicit type annotation to this new module-level constant to satisfy the type-hint requirement for annotatable variables.
Validate the correctness of the flagged issue. If correct, How can I resolve this? If you propose a fix, implement it and please make it concise.
Once fix is implemented, also check other comments on the same PR, and ask user if the user wants to fix the rest of the comments as well. if said yes, then fetch all the comments validate the correctness and implement a minimal fix
```
</details>
<a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F41076&comment_hash=edc8304bdf3a37835f8654736a2dde9e67411c9cf73465024bc7787c318da81e&reaction=like'>👍</a> | <a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F41076&comment_hash=edc8304bdf3a37835f8654736a2dde9e67411c9cf73465024bc7787c318da81e&reaction=dislike'>👎</a>
##########
superset/versioning/activity/windows.py:
##########
@@ -0,0 +1,108 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Pure window arithmetic on half-open ``[start_tx, end_tx)`` intervals.
+
+Extracted from the DB-touching scope resolution so that:
+
+* :mod:`scope` (DB-touching) can import this module at module-top.
+* :mod:`queries.fetch_change_records` can import
+ :func:`row_within_any_window` at module-top instead of through a
+ lazy import that previously dodged a ``scope ↔ queries`` cycle.
+
+Everything here is pure Python — no DB, no Flask. ``end_tx = None``
+means "open-ended (current)" and behaves like positive infinity.
+"""
+
+from __future__ import annotations
+
+from typing import Any
+
+from superset.versioning.activity.kinds import EntityWindows, Window
+
+
+def intersect_windows(outer: Window, inner: Window) -> Window | None:
+ """Intersect two half-open ``[start_tx, end_tx)`` windows.
+
+ Returns the clipped overlap, or ``None`` when they are disjoint.
+ ``end_tx = None`` means "open ended (current)" and acts like
+ positive infinity. Thin wrapper over :meth:`Window.intersect` —
+ kept as a free function so callers and tests don't have to migrate
+ to method form in lockstep with the dataclass promotion.
+ """
+ return outer.intersect(inner)
+
+
+def row_within_any_window(row: dict[str, Any], windows: list[Window]) -> bool:
+ """``True`` iff ``row['transaction_id']`` falls inside at least one
+ of *windows*. Half-open interval semantics match
+ :func:`intersect_windows`."""
+ if not windows:
+ return False
+ tx_id = row["transaction_id"]
Review Comment:
**Suggestion:** Add an explicit type annotation for this local variable to keep relevant values typed instead of leaving them inferred from an Any-typed dictionary lookup. [custom_rule]
**Severity Level:** Minor 🧹
<details>
<summary><b>Why it matters? ⭐ </b></summary>
The function already has precise types for its parameters, and this local value is a relevant variable derived from a dictionary lookup that can be annotated as an int. The custom rule requires type hints on relevant variables that can be annotated, so this is a real violation.
</details>
<details>
<summary><b>Rule source 📖 </b></summary>
.cursor/rules/dev-standard.mdc (line 28)
</details>
[](https://app.codeant.ai/fix-in-ide?tool=cursor&prompt_id=d7e93b8dfd4e4733b5cc3343d6c2fa20&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset) [](https://app.codeant.ai/fix-in-ide?tool=vscode-claude&prompt_id=d7e93b8dfd4e4733b5cc3343d6c2fa20&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset)
*(Use Cmd/Ctrl + Click for best experience)*
<details>
<summary><b>Prompt for AI Agent 🤖 </b></summary>
```mdx
This is a comment left during a code review.
**Path:** superset/versioning/activity/windows.py
**Line:** 55:55
**Comment:**
*Custom Rule: Add an explicit type annotation for this local variable to keep relevant values typed instead of leaving them inferred from an Any-typed dictionary lookup.
Validate the correctness of the flagged issue. If correct, How can I resolve this? If you propose a fix, implement it and please make it concise.
Once fix is implemented, also check other comments on the same PR, and ask user if the user wants to fix the rest of the comments as well. if said yes, then fetch all the comments validate the correctness and implement a minimal fix
```
</details>
<a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F41076&comment_hash=8876267b94993b9d42b8167997efe9d732813cb95f0da5095f2601c526fa6bf4&reaction=like'>👍</a> | <a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F41076&comment_hash=8876267b94993b9d42b8167997efe9d732813cb95f0da5095f2601c526fa6bf4&reaction=dislike'>👎</a>
##########
tests/integration_tests/versioning/change_records_tests.py:
##########
@@ -0,0 +1,715 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Integration tests for ``version_changes`` capture (T052, partial).
+
+Covers in this file:
+ (a) saving a chart with three field changes produces three rows
+ (f) baseline / INSERT transactions produce zero records *for that entity*
+ + unchanged-save / dashboard / params-classification cases
+
+Deferred:
+ (b) ``GET /versions/`` response includes ``changes`` array — lands with
+ T050 (API integration).
+ (c) FK cascade — exercisable in principle (the migration declares
+ ``ON DELETE CASCADE``) but can't be isolated in a unit-style test
+ because ``version_transaction`` is referenced by non-cascading FKs
+ from slices_version / dashboards_version / etc. Covered instead
+ by (d) below once it lands, and by the structural declaration in
+ T046's migration.
+ (d) retention prune drops change records alongside the pruned
+ version — will land when T049 extends ``VersionDAO.prune_versions``
+ to include ``version_changes`` alongside the shadow-row delete.
+ (e) ``kind`` index query plan on Postgres — deferred to T053 perf
+ validation.
+"""
+
+from __future__ import annotations
+
+from datetime import datetime, timedelta
+from typing import Any
+from uuid import uuid4
+
+import pytest
+import sqlalchemy as sa
+from sqlalchemy_continuum import version_class
+
+from superset.extensions import db
+from superset.models.dashboard import Dashboard
+from superset.models.slice import Slice
+from superset.utils import json as _json
+from tests.integration_tests.base_tests import SupersetTestCase
+from tests.integration_tests.fixtures.birth_names_dashboard import ( # noqa: F401
+ load_birth_names_dashboard_with_slices,
+ load_birth_names_data,
+)
+
+_VERSION_CHANGES = sa.table(
+ "version_changes",
+ sa.column("id"),
+ sa.column("transaction_id"),
+ sa.column("entity_kind"),
+ sa.column("entity_id"),
+ sa.column("sequence"),
+ sa.column("kind"),
+ sa.column("operation"),
+ sa.column("path"),
+ sa.column("from_value"),
+ sa.column("to_value"),
+)
+
+_VERSION_TRANSACTION = sa.table(
+ "version_transaction",
+ sa.column("id"),
+ sa.column("issued_at"),
+ sa.column("user_id"),
+ sa.column("action_kind"),
+)
+
+
+def _action_kind_for(tx_id: int) -> str | None:
+ """Read the ``action_kind`` column from the version_transaction row."""
+ return (
+ db.session.connection()
+ .execute(
+ sa.select(_VERSION_TRANSACTION.c.action_kind).where(
+ _VERSION_TRANSACTION.c.id == tx_id
+ )
+ )
+ .scalar()
+ )
+
+
+def _change_rows_for(
+ tx_id: int,
+ *,
+ entity_kind: str | None = None,
+ entity_id: int | None = None,
+) -> list[dict[str, Any]]:
+ """Raw fetch of ``version_changes`` rows for a tx + optional entity filter."""
+ query = sa.select(_VERSION_CHANGES).where(
+ _VERSION_CHANGES.c.transaction_id == tx_id
+ )
+ if entity_kind is not None:
+ query = query.where(_VERSION_CHANGES.c.entity_kind == entity_kind)
+ if entity_id is not None:
+ query = query.where(_VERSION_CHANGES.c.entity_id == entity_id)
+ query = query.order_by(_VERSION_CHANGES.c.sequence.asc())
+ result = db.session.connection().execute(query)
+ return [dict(row._mapping) for row in result]
+
+
+def _persist_fixture_state() -> None:
+ """Commit fixture INSERTs so the baseline row exists before the test edits.
+
+ Without this, the test's first commit batches the fixture's pending
+ INSERTs with the test's UPDATE into a single Continuum transaction
+ and no diff records are emitted (no pre-state).
+ """
+ db.session.commit()
+
+
+class TestChartChangeRecords(SupersetTestCase):
+ """Change-record capture for chart (Slice) saves."""
+
+ @pytest.fixture(autouse=True)
Review Comment:
**Suggestion:** Add type annotations to this fixture helper method by annotating its fixture parameter and explicitly declaring a return type. [custom_rule]
**Severity Level:** Minor 🧹
<details>
<summary><b>Why it matters? ⭐ </b></summary>
This new fixture helper omits type hints on both the `load_birth_names_dashboard_with_slices` parameter and the return type, which matches the custom rule requiring type annotations on new or modified Python functions and methods.
</details>
<details>
<summary><b>Rule source 📖 </b></summary>
.cursor/rules/dev-standard.mdc (line 28)
</details>
[](https://app.codeant.ai/fix-in-ide?tool=cursor&prompt_id=db48b054cc444d4c92297ef3e39e0383&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset) [](https://app.codeant.ai/fix-in-ide?tool=vscode-claude&prompt_id=db48b054cc444d4c92297ef3e39e0383&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset)
*(Use Cmd/Ctrl + Click for best experience)*
<details>
<summary><b>Prompt for AI Agent 🤖 </b></summary>
```mdx
This is a comment left during a code review.
**Path:** tests/integration_tests/versioning/change_records_tests.py
**Line:** 128:128
**Comment:**
*Custom Rule: Add type annotations to this fixture helper method by annotating its fixture parameter and explicitly declaring a return type.
Validate the correctness of the flagged issue. If correct, How can I resolve this? If you propose a fix, implement it and please make it concise.
Once fix is implemented, also check other comments on the same PR, and ask user if the user wants to fix the rest of the comments as well. if said yes, then fetch all the comments validate the correctness and implement a minimal fix
```
</details>
<a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F41076&comment_hash=aa266c99ef3174fbe318c3c81cf8b2215f9a4dec81fc6a19d359e3e470de0ded&reaction=like'>👍</a> | <a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F41076&comment_hash=aa266c99ef3174fbe318c3c81cf8b2215f9a4dec81fc6a19d359e3e470de0ded&reaction=dislike'>👎</a>
##########
tests/integration_tests/versioning/perf_validation_tests.py:
##########
@@ -0,0 +1,416 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""T044 — Performance validation for entity version history.
+
+Skipped by default. Run on demand:
+
+ SUPERSET_PERF_VALIDATION=1 pytest \
+ tests/integration_tests/versioning/perf_validation_tests.py -v -s
+
+Measures the three success criteria defined in the spec:
+
+ * SC-002: version list endpoint responds in under 1 second
+ * SC-004: save path p95 overhead under 50 ms with Continuum tracking
+ on vs. off (FR-014)
+
+(SC-003, the restore-endpoint benchmark, lands with the restore endpoint
+in a later PR.)
+
+The test prints a summary table suitable for pasting into the PR
+description. It also asserts each target so regressions fail loudly
+when the harness is re-run.
+"""
+
+from __future__ import annotations
+
+import os
+import statistics
+import time
+from typing import Any
+
+import pytest
+import sqlalchemy as sa
+from sqlalchemy_continuum import version_class, versioning_manager
+
+from superset.extensions import db
+from superset.models.slice import Slice
+from tests.integration_tests.base_tests import SupersetTestCase
+from tests.integration_tests.constants import ADMIN_USERNAME
+from tests.integration_tests.fixtures.birth_names_dashboard import ( # noqa: F401
+ load_birth_names_dashboard_with_slices,
+ load_birth_names_data,
+)
+
+SKIP_REASON = "Performance validation is manual. Set SUPERSET_PERF_VALIDATION=1 to run."
+
+# Thresholds from spec.md §Success Criteria.
+LIST_ENDPOINT_MAX_MS = 1000 # SC-002
+SAVE_OVERHEAD_P95_MAX_MS = 50 # SC-004
+
+# Activity-view thresholds (sc-107283 §Success Criteria).
+ACTIVITY_ENDPOINT_P95_MAX_MS = 1500 # SC-AV-001
+
+
+def _save_chart_once(chart: Slice, suffix: str) -> None:
+ """One ORM-level save path, mimicking what ChartDAO.update does."""
+ chart.slice_name = f"{chart.slice_name[:64]}_{suffix}"
+ db.session.commit()
+
+
+def _timings_ms(seconds: list[float]) -> dict[str, float]:
+ ms = sorted(s * 1000.0 for s in seconds)
+ return {
+ "p50": statistics.median(ms),
+ "p95": ms[int(len(ms) * 0.95) - 1] if len(ms) >= 20 else max(ms),
+ "max": max(ms),
+ "n": len(ms),
+ }
+
+
+@pytest.mark.skipif(
+ not os.environ.get("SUPERSET_PERF_VALIDATION"),
+ reason=SKIP_REASON,
+)
+class PerfValidationTests(SupersetTestCase):
+ """Runs only when SUPERSET_PERF_VALIDATION=1 is set."""
+
+ @pytest.fixture(autouse=True)
+ def _load_data(self, load_birth_names_dashboard_with_slices: Any) -> None: # noqa: F811, PT004
+ pass
+
+ def _seed_chart_with_n_versions(self, n: int) -> Slice:
+ """Save a chart N times to produce N version rows."""
+ chart = db.session.query(Slice).first()
+ assert chart is not None, "birth_names fixture should provide charts"
+
+ for i in range(n):
+ _save_chart_once(chart, f"v{i}")
+ db.session.commit()
+ return chart
+
+ def test_sc002_list_endpoint_under_1s(self) -> None:
+ """SC-002: list endpoint responds in under 1 second."""
+ self.login(ADMIN_USERNAME)
+
+ # Generate enough versions to exercise the retention-capped state.
+ chart = self._seed_chart_with_n_versions(24)
+ chart_uuid = str(chart.uuid)
+ url = f"/api/v1/chart/{chart_uuid}/versions/"
+
+ # Warm up the endpoint once (JIT caching, mapper configuration, etc.)
+ self.client.get(url)
+
+ timings: list[float] = []
+ for _ in range(10):
+ t0 = time.perf_counter()
+ response = self.client.get(url)
+ timings.append(time.perf_counter() - t0)
+ assert response.status_code == 200
+
+ stats = _timings_ms(timings)
+ print(
+ f"\n[SC-002] GET /versions/ (24 versions) "
+ f"p50={stats['p50']:.1f}ms p95={stats['p95']:.1f}ms "
+ f"max={stats['max']:.1f}ms n={stats['n']}"
+ )
+ assert stats["p95"] < LIST_ENDPOINT_MAX_MS, (
+ f"SC-002 failed: list endpoint p95 {stats['p95']:.1f}ms "
+ f">= {LIST_ENDPOINT_MAX_MS}ms"
+ )
+
+ # SC-003 (restore endpoint under 3s) intentionally omitted here: the
+ # version-restore route (POST /<uuid>/versions/<version_uuid>/restore)
+ # ships in a later PR, so there is nothing to time at this point in the
+ # stack. Re-add this benchmark alongside the restore endpoint.
+
+ def test_sc004_save_overhead_under_50ms(self) -> None:
+ """SC-004: save path p95 overhead under 50ms (FR-014).
+
+ Toggling Continuum on and off mid-process corrupts its internal
+ ``units_of_work`` state and is not a reliable measurement. Instead
+ this test directly measures the wall-clock time spent inside the
+ four session-level listeners Continuum attaches to
+ ``sa.orm.session.Session`` — ``before_flush``, ``after_flush``,
+ ``after_commit``, ``after_rollback`` — plus Superset's own
+ baseline / snapshot / retention-prune listeners (attached to
+ ``db.session``). The cumulative listener time per save is the
+ marginal overhead version capture adds over a save with
+ versioning removed entirely, because without these listeners
+ the ORM would not execute any of that code.
+
+ The approach:
+ 1. Wrap each known listener with a timing proxy that adds its
+ wall-clock time to a per-save accumulator.
+ 2. Save the same chart N times, recording each save's
+ accumulator value.
+ 3. Compute p50 / p95 of the per-save overhead.
+
+ This matches the measurement intent of SC-004 (how much does
+ versioning cost per save) without the fragility of toggling
+ Continuum mid-test.
+ """
+ self.login(ADMIN_USERNAME)
+
+ chart = db.session.query(Slice).first()
+ assert chart is not None
+
+ # Per-save accumulator incremented by the wrapped listeners.
+ acc = [0.0]
+
+ def wrap_listener(original: Any) -> Any:
+ def wrapper(*args: Any, **kwargs: Any) -> Any:
+ t0 = time.perf_counter()
+ try:
+ return original(*args, **kwargs)
+ finally:
+ acc[0] += time.perf_counter() - t0
+
+ wrapper.__wrapped__ = original # type: ignore[attr-defined]
+ return wrapper
+
+ # Instrument Continuum's four session listeners by detaching the
+ # bound method, wrapping, and re-attaching under a single-use
+ # listener handle we can cleanly remove on teardown.
+ session_target = sa.orm.session.Session
+ attached: list[tuple[str, Any]] = []
+ for event_name, listener in list(versioning_manager.session_listeners.items()):
+ sa.event.remove(session_target, event_name, listener)
+ wrapped = wrap_listener(listener)
+ sa.event.listen(session_target, event_name, wrapped)
+ attached.append((event_name, wrapped))
+
+ iterations = 100
+ warmup = 5
+ try:
+ # Warmup (first baseline INSERT, JIT, cache warming).
+ for i in range(warmup):
+ _save_chart_once(chart, f"warm_{i}")
+ acc[0] = 0.0
+
+ total_timings: list[float] = []
+ overhead_timings: list[float] = []
+ for i in range(iterations):
+ acc[0] = 0.0
+ t0 = time.perf_counter()
+ _save_chart_once(chart, f"run_{i}")
+ total_timings.append(time.perf_counter() - t0)
+ overhead_timings.append(acc[0])
+ finally:
+ for event_name, wrapped in attached:
+ sa.event.remove(session_target, event_name, wrapped)
+ sa.event.listen(
+ session_target,
+ event_name,
+ wrapped.__wrapped__,
+ )
+
+ total = _timings_ms(total_timings)
+ overhead = _timings_ms(overhead_timings)
+
+ ver_cls = version_class(Slice)
+ produced = db.session.query(ver_cls).filter(ver_cls.id == chart.id).count()
+ print(
+ f"\n[SC-004] save iterations={iterations} chart_id={chart.id} "
+ f"version_rows_produced={produced}"
+ )
+ print(
+ f"[SC-004] full save: "
+ f"p50={total['p50']:.2f}ms p95={total['p95']:.2f}ms "
+ f"max={total['max']:.2f}ms"
+ )
+ print(
+ f"[SC-004] version-cap overhead: "
+ f"p50={overhead['p50']:.2f}ms p95={overhead['p95']:.2f}ms "
+ f"max={overhead['max']:.2f}ms"
+ )
+
+ assert overhead["p95"] < SAVE_OVERHEAD_P95_MAX_MS, (
+ f"SC-004 failed: version-capture p95 overhead "
+ f"{overhead['p95']:.2f}ms >= {SAVE_OVERHEAD_P95_MAX_MS}ms"
+ )
+
+ # ---- T045: Activity-view perf validation -----------------------------
+
+ def _seed_activity_history(self) -> str:
+ """Generate dense history on the birth_names dashboard so the
+ activity endpoint has something realistic to read.
+
+ T045's spec target is "25 charts × 3 dataset windows each". The
+ birth_names fixture has ~12 charts on a single dataset (no
+ multi-dataset support without a bespoke fixture). We approximate
+ the load by: (a) editing many charts on the dashboard, (b)
+ editing the dataset's description several times, (c) editing the
+ dashboard's own title once. That yields ~30+ change records
+ spanning all three entity kinds — enough to exercise the
+ decoration, visibility, and impact-batch paths without needing a
+ multi-dataset fixture builder. Returns the dashboard UUID.
+
+ **Why this commits without rollback** (unlike the test bodies in
+ ``activity_view_tests.py``): the whole point of a perf seed is
+ that the rows it produces have actually been persisted, so the
+ endpoint hit that follows reads a realistic state of the
+ ``version_changes`` / shadow tables. T053's
+ ``try/finally``+``rollback`` convention is for tests that
+ assert on *which records were captured*; here the seed IS the
+ setup, not the unit under test. The fixture's session-scoped
+ ``_cleanup`` removes the dashboard / slices at session teardown,
+ which is when the shadow rows age out too.
+ """
+ # pylint: disable=import-outside-toplevel
+ from superset.connectors.sqla.models import SqlaTable
+ from superset.models.dashboard import Dashboard
+
+ dashboard = (
+ db.session.query(Dashboard)
+ .filter(Dashboard.dashboard_title.like("USA Births%"))
+ .first()
+ )
Review Comment:
**Suggestion:** Add an explicit optional model type annotation to this query result variable. [custom_rule]
**Severity Level:** Minor 🧹
<details>
<summary><b>Why it matters? ⭐ </b></summary>
The query result is assigned to a variable without an explicit type hint even though it is potentially nullable. This is a relevant variable that can be annotated, so the rule violation is present.
</details>
<details>
<summary><b>Rule source 📖 </b></summary>
.cursor/rules/dev-standard.mdc (line 28)
</details>
[](https://app.codeant.ai/fix-in-ide?tool=cursor&prompt_id=adc9129b353249fb9d9c69baa498d2be&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset) [](https://app.codeant.ai/fix-in-ide?tool=vscode-claude&prompt_id=adc9129b353249fb9d9c69baa498d2be&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset)
*(Use Cmd/Ctrl + Click for best experience)*
<details>
<summary><b>Prompt for AI Agent 🤖 </b></summary>
```mdx
This is a comment left during a code review.
**Path:** tests/integration_tests/versioning/perf_validation_tests.py
**Line:** 277:281
**Comment:**
*Custom Rule: Add an explicit optional model type annotation to this query result variable.
Validate the correctness of the flagged issue. If correct, How can I resolve this? If you propose a fix, implement it and please make it concise.
Once fix is implemented, also check other comments on the same PR, and ask user if the user wants to fix the rest of the comments as well. if said yes, then fetch all the comments validate the correctness and implement a minimal fix
```
</details>
<a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F41076&comment_hash=30070faa7840fc835c91aa4c225f47debf8ed197646206172cf732959eed76a7&reaction=like'>👍</a> | <a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F41076&comment_hash=30070faa7840fc835c91aa4c225f47debf8ed197646206172cf732959eed76a7&reaction=dislike'>👎</a>
##########
tests/integration_tests/versioning/activity_view_tests.py:
##########
@@ -0,0 +1,1250 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Integration tests for the cross-entity activity-view API (sc-107283).
+
+US1 — dashboard activity stream: ``GET /api/v1/dashboard/<uuid>/activity/``.
+Tests for US2 (chart activity) and US3 (dataset activity) come in later
+phases.
+
+Per spec T053 / sc-103156 T062, every test that mutates a fixture entity
+wraps the test body in ``try``/``finally`` with
+``metadata_db.session.rollback()`` in the ``finally``. The rationale is
+documented in the spec — Continuum captures dirty mappers during
+autoflush, so leaving an instrumented attribute dirty pollutes
+downstream tests via the shadow tables.
+"""
+
+from __future__ import annotations
+
+from typing import Any
+from uuid import uuid4
+
+import pytest
+
+from superset.connectors.sqla.models import SqlaTable
+from superset.extensions import db
+from superset.models.dashboard import Dashboard
+from superset.models.slice import Slice
+from superset.utils import json as _json
+from tests.integration_tests.base_tests import SupersetTestCase
+from tests.integration_tests.constants import (
+ ADMIN_USERNAME,
+ ALPHA_USERNAME,
+ GAMMA_USERNAME,
+)
+from tests.integration_tests.fixtures.birth_names_dashboard import ( # noqa: F401
+ load_birth_names_dashboard_with_slices,
+ load_birth_names_data,
+)
+
+
+def _get_birth_names_dataset() -> SqlaTable:
+ return (
+ db.session.query(SqlaTable)
+ .filter(SqlaTable.table_name == "birth_names")
+ .first()
+ )
+
+
+def _persist_fixture_state() -> None:
+ """Force the fixture's pending INSERTs to commit so subsequent edits
+ produce *new* version rows instead of being batched into the
+ creation transaction. Mirrors the same helper in
+ ``tests/integration_tests/dashboards/version_history_tests.py``.
+ """
+ db.session.commit()
+
+
+def _get_birth_names_dashboard() -> Dashboard:
+ return (
+ db.session.query(Dashboard)
+ .filter(Dashboard.dashboard_title == "USA Births Names")
+ .first()
+ )
+
+
+class TestDashboardActivityView(SupersetTestCase):
+ """T017–T026 — ``GET /api/v1/dashboard/<uuid>/activity/`` (US1)."""
+
+ @pytest.fixture(autouse=True)
Review Comment:
**Suggestion:** Add explicit type hints to this fixture method by annotating the fixture parameter and the method return type. [custom_rule]
**Severity Level:** Minor 🧹
<details>
<summary><b>Why it matters? ⭐ </b></summary>
This fixture method omits type hints for its parameter and return type, which matches the custom rule requiring explicit Python type hints on functions/methods where they can be annotated.
</details>
<details>
<summary><b>Rule source 📖 </b></summary>
.cursor/rules/dev-standard.mdc (line 28)
</details>
[](https://app.codeant.ai/fix-in-ide?tool=cursor&prompt_id=65e5db8a3c364965bb385aff4f58a28e&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset) [](https://app.codeant.ai/fix-in-ide?tool=vscode-claude&prompt_id=65e5db8a3c364965bb385aff4f58a28e&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset)
*(Use Cmd/Ctrl + Click for best experience)*
<details>
<summary><b>Prompt for AI Agent 🤖 </b></summary>
```mdx
This is a comment left during a code review.
**Path:** tests/integration_tests/versioning/activity_view_tests.py
**Line:** 83:83
**Comment:**
*Custom Rule: Add explicit type hints to this fixture method by annotating the fixture parameter and the method return type.
Validate the correctness of the flagged issue. If correct, How can I resolve this? If you propose a fix, implement it and please make it concise.
Once fix is implemented, also check other comments on the same PR, and ask user if the user wants to fix the rest of the comments as well. if said yes, then fetch all the comments validate the correctness and implement a minimal fix
```
</details>
<a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F41076&comment_hash=1405618e2b3a3e9e9de2baf99f8c5c43078e881cd6b9c46c58e5f27028ca8224&reaction=like'>👍</a> | <a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F41076&comment_hash=1405618e2b3a3e9e9de2baf99f8c5c43078e881cd6b9c46c58e5f27028ca8224&reaction=dislike'>👎</a>
##########
tests/integration_tests/versioning/perf_validation_tests.py:
##########
@@ -0,0 +1,416 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""T044 — Performance validation for entity version history.
+
+Skipped by default. Run on demand:
+
+ SUPERSET_PERF_VALIDATION=1 pytest \
+ tests/integration_tests/versioning/perf_validation_tests.py -v -s
+
+Measures the three success criteria defined in the spec:
+
+ * SC-002: version list endpoint responds in under 1 second
+ * SC-004: save path p95 overhead under 50 ms with Continuum tracking
+ on vs. off (FR-014)
+
+(SC-003, the restore-endpoint benchmark, lands with the restore endpoint
+in a later PR.)
+
+The test prints a summary table suitable for pasting into the PR
+description. It also asserts each target so regressions fail loudly
+when the harness is re-run.
+"""
+
+from __future__ import annotations
+
+import os
+import statistics
+import time
+from typing import Any
+
+import pytest
+import sqlalchemy as sa
+from sqlalchemy_continuum import version_class, versioning_manager
+
+from superset.extensions import db
+from superset.models.slice import Slice
+from tests.integration_tests.base_tests import SupersetTestCase
+from tests.integration_tests.constants import ADMIN_USERNAME
+from tests.integration_tests.fixtures.birth_names_dashboard import ( # noqa: F401
+ load_birth_names_dashboard_with_slices,
+ load_birth_names_data,
+)
+
+SKIP_REASON = "Performance validation is manual. Set SUPERSET_PERF_VALIDATION=1 to run."
+
+# Thresholds from spec.md §Success Criteria.
+LIST_ENDPOINT_MAX_MS = 1000 # SC-002
+SAVE_OVERHEAD_P95_MAX_MS = 50 # SC-004
+
+# Activity-view thresholds (sc-107283 §Success Criteria).
+ACTIVITY_ENDPOINT_P95_MAX_MS = 1500 # SC-AV-001
+
+
+def _save_chart_once(chart: Slice, suffix: str) -> None:
+ """One ORM-level save path, mimicking what ChartDAO.update does."""
+ chart.slice_name = f"{chart.slice_name[:64]}_{suffix}"
+ db.session.commit()
+
+
+def _timings_ms(seconds: list[float]) -> dict[str, float]:
+ ms = sorted(s * 1000.0 for s in seconds)
+ return {
+ "p50": statistics.median(ms),
+ "p95": ms[int(len(ms) * 0.95) - 1] if len(ms) >= 20 else max(ms),
+ "max": max(ms),
+ "n": len(ms),
+ }
+
+
+@pytest.mark.skipif(
+ not os.environ.get("SUPERSET_PERF_VALIDATION"),
+ reason=SKIP_REASON,
+)
+class PerfValidationTests(SupersetTestCase):
+ """Runs only when SUPERSET_PERF_VALIDATION=1 is set."""
+
+ @pytest.fixture(autouse=True)
+ def _load_data(self, load_birth_names_dashboard_with_slices: Any) -> None: # noqa: F811, PT004
+ pass
+
+ def _seed_chart_with_n_versions(self, n: int) -> Slice:
+ """Save a chart N times to produce N version rows."""
+ chart = db.session.query(Slice).first()
+ assert chart is not None, "birth_names fixture should provide charts"
+
+ for i in range(n):
+ _save_chart_once(chart, f"v{i}")
+ db.session.commit()
+ return chart
+
+ def test_sc002_list_endpoint_under_1s(self) -> None:
+ """SC-002: list endpoint responds in under 1 second."""
+ self.login(ADMIN_USERNAME)
+
+ # Generate enough versions to exercise the retention-capped state.
+ chart = self._seed_chart_with_n_versions(24)
+ chart_uuid = str(chart.uuid)
+ url = f"/api/v1/chart/{chart_uuid}/versions/"
+
+ # Warm up the endpoint once (JIT caching, mapper configuration, etc.)
+ self.client.get(url)
+
+ timings: list[float] = []
+ for _ in range(10):
+ t0 = time.perf_counter()
+ response = self.client.get(url)
+ timings.append(time.perf_counter() - t0)
+ assert response.status_code == 200
+
+ stats = _timings_ms(timings)
+ print(
+ f"\n[SC-002] GET /versions/ (24 versions) "
+ f"p50={stats['p50']:.1f}ms p95={stats['p95']:.1f}ms "
+ f"max={stats['max']:.1f}ms n={stats['n']}"
+ )
+ assert stats["p95"] < LIST_ENDPOINT_MAX_MS, (
+ f"SC-002 failed: list endpoint p95 {stats['p95']:.1f}ms "
+ f">= {LIST_ENDPOINT_MAX_MS}ms"
+ )
+
+ # SC-003 (restore endpoint under 3s) intentionally omitted here: the
+ # version-restore route (POST /<uuid>/versions/<version_uuid>/restore)
+ # ships in a later PR, so there is nothing to time at this point in the
+ # stack. Re-add this benchmark alongside the restore endpoint.
+
+ def test_sc004_save_overhead_under_50ms(self) -> None:
+ """SC-004: save path p95 overhead under 50ms (FR-014).
+
+ Toggling Continuum on and off mid-process corrupts its internal
+ ``units_of_work`` state and is not a reliable measurement. Instead
+ this test directly measures the wall-clock time spent inside the
+ four session-level listeners Continuum attaches to
+ ``sa.orm.session.Session`` — ``before_flush``, ``after_flush``,
+ ``after_commit``, ``after_rollback`` — plus Superset's own
+ baseline / snapshot / retention-prune listeners (attached to
+ ``db.session``). The cumulative listener time per save is the
+ marginal overhead version capture adds over a save with
+ versioning removed entirely, because without these listeners
+ the ORM would not execute any of that code.
+
+ The approach:
+ 1. Wrap each known listener with a timing proxy that adds its
+ wall-clock time to a per-save accumulator.
+ 2. Save the same chart N times, recording each save's
+ accumulator value.
+ 3. Compute p50 / p95 of the per-save overhead.
+
+ This matches the measurement intent of SC-004 (how much does
+ versioning cost per save) without the fragility of toggling
+ Continuum mid-test.
+ """
+ self.login(ADMIN_USERNAME)
+
+ chart = db.session.query(Slice).first()
+ assert chart is not None
+
+ # Per-save accumulator incremented by the wrapped listeners.
+ acc = [0.0]
Review Comment:
**Suggestion:** Add an explicit type annotation for this local accumulator variable to satisfy the typing rule for relevant variables. [custom_rule]
**Severity Level:** Minor 🧹
<details>
<summary><b>Why it matters? ⭐ </b></summary>
This local accumulator is a relevant variable that can be annotated, but it is created without a type hint. That matches the rule requiring type hints on relevant Python variables.
</details>
<details>
<summary><b>Rule source 📖 </b></summary>
.cursor/rules/dev-standard.mdc (line 28)
</details>
[](https://app.codeant.ai/fix-in-ide?tool=cursor&prompt_id=e80fdd1c30f0405aa9654d16e9ec9a0d&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset) [](https://app.codeant.ai/fix-in-ide?tool=vscode-claude&prompt_id=e80fdd1c30f0405aa9654d16e9ec9a0d&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset)
*(Use Cmd/Ctrl + Click for best experience)*
<details>
<summary><b>Prompt for AI Agent 🤖 </b></summary>
```mdx
This is a comment left during a code review.
**Path:** tests/integration_tests/versioning/perf_validation_tests.py
**Line:** 172:172
**Comment:**
*Custom Rule: Add an explicit type annotation for this local accumulator variable to satisfy the typing rule for relevant variables.
Validate the correctness of the flagged issue. If correct, How can I resolve this? If you propose a fix, implement it and please make it concise.
Once fix is implemented, also check other comments on the same PR, and ask user if the user wants to fix the rest of the comments as well. if said yes, then fetch all the comments validate the correctness and implement a minimal fix
```
</details>
<a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F41076&comment_hash=bab4f013ce2c2ca29ea4d61d14287e16b6def7e610c8e47d5dc6f83b7dfe2ef6&reaction=like'>👍</a> | <a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F41076&comment_hash=bab4f013ce2c2ca29ea4d61d14287e16b6def7e610c8e47d5dc6f83b7dfe2ef6&reaction=dislike'>👎</a>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(versioning): cross-entity version activity view [superset]
Posted by "codeant-ai-for-open-source[bot] (via GitHub)" <gi...@apache.org>.
codeant-ai-for-open-source[bot] commented on code in PR #41076:
URL: https://github.com/apache/superset/pull/41076#discussion_r3500178524
##########
scripts/seed_junction_load.py:
##########
@@ -0,0 +1,679 @@
+#!/usr/bin/env python3
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+# ----------------------------------------------------------------------
+# Stress-test data generator for the composite-PK migration (sc-105349).
+#
+# Bulk-inserts synthetic parent rows and many-to-many junction rows for
+# the eight association tables that the composite-PK migration touches.
+# Useful for measuring migration runtime at varying scales — run this at
+# 100K / 1M / 5M / 10M rows and time the migration at each scale to
+# verify the O(N log N) extrapolation.
+#
+# Idempotent: rerunning with the same target is a no-op; rerunning with
+# a higher target adds rows up to the new total. Batched bulk INSERTs
+# (10K rows per statement) make it fast on Postgres, MySQL, and SQLite.
+#
+# Usage (inside the Superset container):
+#
+# docker exec superset-superset-1 \\
+# /app/.venv/bin/python /app/scripts/seed_junction_load.py \\
+# --dashboard-slices 1000000 \\
+# --slice-user 100000 \\
+# --dashboard-user 100000
+#
+# Run with no flags for the defaults shown below. Use ``--dry-run`` to
+# print the planned inserts without writing anything.
+#
+# The script connects via Superset's standard ``DATABASE_*`` env vars
+# (or ``SUPERSET__SQLALCHEMY_DATABASE_URI`` if set), so it works
+# automatically inside the Superset container regardless of which
+# metadata DB backend is in use.
+
+from __future__ import annotations
+
+import argparse
+import logging
+import os
+import sys
+import time
+from contextlib import contextmanager
+from typing import Iterator
+from uuid import uuid4
+
+import sqlalchemy as sa
+from sqlalchemy.engine import Connection, Engine
+
+logger = logging.getLogger("seed_junction_load")
+
+# Bulk INSERT batch size. Larger values = fewer statements but more memory.
+BATCH = 10_000
+
+# Default per-junction-table target row counts. Tuned to mimic the shape
+# of a large multi-team Superset install. Override via CLI flags.
+DEFAULTS: dict[str, int] = {
+ "dashboard_slices": 1_000_000,
+ "slice_user": 100_000,
+ "dashboard_user": 100_000,
+ "dashboard_roles": 10_000,
+}
+
+# (junction_table, fk1_col, fk2_col, parent1_table, parent2_table)
+# parents reference id columns; we generate (fk1, fk2) pairs by sampling
+# from the parents' existing IDs.
+JUNCTIONS: list[tuple[str, str, str, str, str]] = [
+ ("dashboard_slices", "dashboard_id", "slice_id", "dashboards", "slices"),
+ ("slice_user", "user_id", "slice_id", "ab_user", "slices"),
+ ("dashboard_user", "user_id", "dashboard_id", "ab_user", "dashboards"),
+ ("dashboard_roles", "dashboard_id", "role_id", "dashboards", "ab_role"),
+]
+
+# Junction tables that originally carried ``UNIQUE(fk1, fk2)`` and therefore
+# cannot accept duplicate ``(fk1, fk2)`` pairs even on the pre-migration
+# (downgrade) schema. The other JUNCTIONS allow duplicates pre-migration.
+# Only ``dashboard_slices`` is listed: the migration's other UNIQUE table
+# (``report_schedule_user``) is not in JUNCTIONS — this script doesn't seed
+# it — so listing it here would imply coverage that doesn't exist. Add it
+# alongside a JUNCTIONS entry if that table ever gets seeded.
+JUNCTIONS_WITH_UNIQUE: set[str] = {"dashboard_slices"}
+
+
+# ----------------------------------------------------------------------
+# Connection setup
+# ----------------------------------------------------------------------
+
+
+def build_engine() -> Engine:
+ """Build a SQLAlchemy engine from Superset env vars."""
+ if uri := os.environ.get("SUPERSET__SQLALCHEMY_DATABASE_URI"):
+ logger.info("Using SUPERSET__SQLALCHEMY_DATABASE_URI from env")
+ return sa.create_engine(uri)
+
+ try:
+ dialect = os.environ["DATABASE_DIALECT"]
+ user = os.environ["DATABASE_USER"]
+ password = os.environ["DATABASE_PASSWORD"]
+ host = os.environ["DATABASE_HOST"]
+ port = os.environ["DATABASE_PORT"]
+ db = os.environ["DATABASE_DB"]
+ except KeyError as exc:
+ sys.exit(
+ f"Missing env var {exc}; either set DATABASE_DIALECT/USER/PASSWORD/"
+ f"HOST/PORT/DB or SUPERSET__SQLALCHEMY_DATABASE_URI before running."
+ )
+
+ uri = f"{dialect}://{user}:{password}@{host}:{port}/{db}"
Review Comment:
**Suggestion:** The connection URI is built via raw string interpolation, so special characters in credentials (for example `@`, `:`, `/`, `#`) will be parsed as URL delimiters and break authentication. Build the engine URL with SQLAlchemy URL helpers (or URL-encode each component) to avoid runtime connection failures in real environments. [api mismatch]
<details>
<summary><b>Severity Level:</b> Major ⚠️</summary>
```mdx
- ❌ Seed script may fail to connect to metadata DB.
- ⚠️ Large-scale migration load tests become impossible with complex passwords.
- ⚠️ Operators may weaken passwords to make script work.
```
</details>
<details>
<summary><b>Steps of Reproduction ✅ </b></summary>
```mdx
1. In `scripts/seed_junction_load.py:101-124`, observe `build_engine()` builds a
SQLAlchemy engine by first checking `SUPERSET__SQLALCHEMY_DATABASE_URI` and, if unset,
reading `DATABASE_DIALECT`, `DATABASE_USER`, `DATABASE_PASSWORD`, `DATABASE_HOST`,
`DATABASE_PORT`, and `DATABASE_DB` from the environment, then interpolating them into `uri
= f"{dialect}://{user}:{password}@{host}:{port}/{db}"` at line 120 and passing that to
`sa.create_engine(uri)`.
2. In the same file at `run()` (`scripts/seed_junction_load.py:196-211`), verify that this
helper is the only engine constructor: `run()` calls `engine = build_engine()` and uses
`with engine.begin() as conn:` to open a connection for all seeding work.
3. In `main()` (`scripts/seed_junction_load.py:218-280`), confirm that the CLI entrypoint
ultimately calls `run(targets, dry_run=args.dry_run,
dirty_duplicates_pct=args.dirty_duplicates_pct)` inside a `time_phase("total")` block, so
any invocation of the script without `SUPERSET__SQLALCHEMY_DATABASE_URI` set will exercise
the interpolated URI path.
4. Start the Superset container (or run the script in a similar environment) with
`SUPERSET__SQLALCHEMY_DATABASE_URI` unset, set `DATABASE_DIALECT` (for example
`postgresql`), `DATABASE_HOST`, `DATABASE_PORT`, `DATABASE_DB`, and set
`DATABASE_PASSWORD` to a value containing URL-reserved characters such as `P@ss#word/1`;
then run `/app/.venv/bin/python /app/scripts/seed_junction_load.py` as documented in the
header (lines 32-38) and observe that SQLAlchemy’s `create_engine` fails to parse or
authenticate against the database because the raw f-string in `build_engine()` treats `@`,
`:`, `#`, or `/` as URI delimiters instead of as literal password characters.
```
</details>
[](https://app.codeant.ai/fix-in-ide?tool=cursor&prompt_id=197aac905320400598312ed198e30ca0&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset) [](https://app.codeant.ai/fix-in-ide?tool=vscode-claude&prompt_id=197aac905320400598312ed198e30ca0&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset)
*(Use Cmd/Ctrl + Click for best experience)*
<details>
<summary><b>Prompt for AI Agent 🤖 </b></summary>
```mdx
This is a comment left during a code review.
**Path:** scripts/seed_junction_load.py
**Line:** 120:120
**Comment:**
*Api Mismatch: The connection URI is built via raw string interpolation, so special characters in credentials (for example `@`, `:`, `/`, `#`) will be parsed as URL delimiters and break authentication. Build the engine URL with SQLAlchemy URL helpers (or URL-encode each component) to avoid runtime connection failures in real environments.
Validate the correctness of the flagged issue. If correct, How can I resolve this? If you propose a fix, implement it and please make it concise.
Once fix is implemented, also check other comments on the same PR, and ask user if the user wants to fix the rest of the comments as well. if said yes, then fetch all the comments validate the correctness and implement a minimal fix
```
</details>
<a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F41076&comment_hash=53a2a41f5b67614626d4a9ad8a3de888c93e379f8d4724189efc43a78109c566&reaction=like'>👍</a> | <a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F41076&comment_hash=53a2a41f5b67614626d4a9ad8a3de888c93e379f8d4724189efc43a78109c566&reaction=dislike'>👎</a>
##########
scripts/seed_junction_load.py:
##########
@@ -0,0 +1,679 @@
+#!/usr/bin/env python3
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+# ----------------------------------------------------------------------
+# Stress-test data generator for the composite-PK migration (sc-105349).
+#
+# Bulk-inserts synthetic parent rows and many-to-many junction rows for
+# the eight association tables that the composite-PK migration touches.
Review Comment:
**Suggestion:** The header comment states the script seeds eight association tables, but this implementation only defines four seeded junction tables in `JUNCTIONS`. Update the comment to match actual behavior so operators do not run incorrect capacity/performance tests based on wrong table coverage. [comment mismatch]
<details>
<summary><b>Severity Level:</b> Minor 🧹</summary>
```mdx
- ⚠️ Header comment misstates junction-table coverage for operators.
- ⚠️ Risk of overestimating migration test coverage from script.
- ⚠️ Suggestion purely documentation; runtime behavior unaffected.
```
</details>
<details>
<summary><b>Steps of Reproduction ✅ </b></summary>
```mdx
1. Open `scripts/seed_junction_load.py` and look at the module header around lines 20-26:
the comment at lines 22-23 states that the script “Bulk-inserts synthetic parent rows and
many-to-many junction rows for the eight association tables that the composite-PK
migration touches.”
2. Scroll down to the `JUNCTIONS` constant at lines 76-84, which defines the junctions the
script actually seeds as a list of four tuples: `dashboard_slices`, `slice_user`,
`dashboard_user`, and `dashboard_roles`.
3. Inspect `_seed_all_junctions` at lines 97-105 and verify that it iterates directly over
`JUNCTIONS` and calls `seed_junction()` per entry, so only those four association tables
are ever seeded by this script.
4. Examine `DEFAULTS` at lines 67-74 and confirm that the CLI only exposes target row
counts for the same four junction tables; there is no configuration or code path seeding
additional association tables, so the “eight association tables” statement in the header
comment is inaccurate documentation rather than reflecting current behavior.
```
</details>
[](https://app.codeant.ai/fix-in-ide?tool=cursor&prompt_id=28b46c357424420cbdeaed4b9edd0976&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset) [](https://app.codeant.ai/fix-in-ide?tool=vscode-claude&prompt_id=28b46c357424420cbdeaed4b9edd0976&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset)
*(Use Cmd/Ctrl + Click for best experience)*
<details>
<summary><b>Prompt for AI Agent 🤖 </b></summary>
```mdx
This is a comment left during a code review.
**Path:** scripts/seed_junction_load.py
**Line:** 22:23
**Comment:**
*Comment Mismatch: The header comment states the script seeds eight association tables, but this implementation only defines four seeded junction tables in `JUNCTIONS`. Update the comment to match actual behavior so operators do not run incorrect capacity/performance tests based on wrong table coverage.
Validate the correctness of the flagged issue. If correct, How can I resolve this? If you propose a fix, implement it and please make it concise.
Once fix is implemented, also check other comments on the same PR, and ask user if the user wants to fix the rest of the comments as well. if said yes, then fetch all the comments validate the correctness and implement a minimal fix
```
</details>
<a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F41076&comment_hash=41f007c3895afd927dc909c013ffbaf984acebbfa03b7a71dcb38e5a40135744&reaction=like'>👍</a> | <a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F41076&comment_hash=41f007c3895afd927dc909c013ffbaf984acebbfa03b7a71dcb38e5a40135744&reaction=dislike'>👎</a>
##########
scripts/seed_junction_load.py:
##########
@@ -0,0 +1,679 @@
+#!/usr/bin/env python3
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+# ----------------------------------------------------------------------
+# Stress-test data generator for the composite-PK migration (sc-105349).
+#
+# Bulk-inserts synthetic parent rows and many-to-many junction rows for
+# the eight association tables that the composite-PK migration touches.
+# Useful for measuring migration runtime at varying scales — run this at
+# 100K / 1M / 5M / 10M rows and time the migration at each scale to
+# verify the O(N log N) extrapolation.
+#
+# Idempotent: rerunning with the same target is a no-op; rerunning with
+# a higher target adds rows up to the new total. Batched bulk INSERTs
+# (10K rows per statement) make it fast on Postgres, MySQL, and SQLite.
+#
+# Usage (inside the Superset container):
+#
+# docker exec superset-superset-1 \\
+# /app/.venv/bin/python /app/scripts/seed_junction_load.py \\
+# --dashboard-slices 1000000 \\
+# --slice-user 100000 \\
+# --dashboard-user 100000
+#
+# Run with no flags for the defaults shown below. Use ``--dry-run`` to
+# print the planned inserts without writing anything.
+#
+# The script connects via Superset's standard ``DATABASE_*`` env vars
+# (or ``SUPERSET__SQLALCHEMY_DATABASE_URI`` if set), so it works
+# automatically inside the Superset container regardless of which
+# metadata DB backend is in use.
+
+from __future__ import annotations
+
+import argparse
+import logging
+import os
+import sys
+import time
+from contextlib import contextmanager
+from typing import Iterator
+from uuid import uuid4
+
+import sqlalchemy as sa
+from sqlalchemy.engine import Connection, Engine
+
+logger = logging.getLogger("seed_junction_load")
+
+# Bulk INSERT batch size. Larger values = fewer statements but more memory.
+BATCH = 10_000
+
+# Default per-junction-table target row counts. Tuned to mimic the shape
+# of a large multi-team Superset install. Override via CLI flags.
+DEFAULTS: dict[str, int] = {
+ "dashboard_slices": 1_000_000,
+ "slice_user": 100_000,
+ "dashboard_user": 100_000,
+ "dashboard_roles": 10_000,
+}
+
+# (junction_table, fk1_col, fk2_col, parent1_table, parent2_table)
+# parents reference id columns; we generate (fk1, fk2) pairs by sampling
+# from the parents' existing IDs.
+JUNCTIONS: list[tuple[str, str, str, str, str]] = [
+ ("dashboard_slices", "dashboard_id", "slice_id", "dashboards", "slices"),
+ ("slice_user", "user_id", "slice_id", "ab_user", "slices"),
+ ("dashboard_user", "user_id", "dashboard_id", "ab_user", "dashboards"),
+ ("dashboard_roles", "dashboard_id", "role_id", "dashboards", "ab_role"),
+]
+
+# Junction tables that originally carried ``UNIQUE(fk1, fk2)`` and therefore
+# cannot accept duplicate ``(fk1, fk2)`` pairs even on the pre-migration
+# (downgrade) schema. The other JUNCTIONS allow duplicates pre-migration.
+# Only ``dashboard_slices`` is listed: the migration's other UNIQUE table
+# (``report_schedule_user``) is not in JUNCTIONS — this script doesn't seed
+# it — so listing it here would imply coverage that doesn't exist. Add it
+# alongside a JUNCTIONS entry if that table ever gets seeded.
+JUNCTIONS_WITH_UNIQUE: set[str] = {"dashboard_slices"}
+
+
+# ----------------------------------------------------------------------
+# Connection setup
+# ----------------------------------------------------------------------
+
+
+def build_engine() -> Engine:
+ """Build a SQLAlchemy engine from Superset env vars."""
+ if uri := os.environ.get("SUPERSET__SQLALCHEMY_DATABASE_URI"):
+ logger.info("Using SUPERSET__SQLALCHEMY_DATABASE_URI from env")
+ return sa.create_engine(uri)
+
+ try:
+ dialect = os.environ["DATABASE_DIALECT"]
+ user = os.environ["DATABASE_USER"]
+ password = os.environ["DATABASE_PASSWORD"]
+ host = os.environ["DATABASE_HOST"]
+ port = os.environ["DATABASE_PORT"]
+ db = os.environ["DATABASE_DB"]
+ except KeyError as exc:
+ sys.exit(
+ f"Missing env var {exc}; either set DATABASE_DIALECT/USER/PASSWORD/"
+ f"HOST/PORT/DB or SUPERSET__SQLALCHEMY_DATABASE_URI before running."
+ )
+
+ uri = f"{dialect}://{user}:{password}@{host}:{port}/{db}"
+ logger.info(
+ "Built URI from DATABASE_* env vars (dialect=%s, host=%s)", dialect, host
+ )
+ return sa.create_engine(uri)
+
+
+# ----------------------------------------------------------------------
+# Helpers
+# ----------------------------------------------------------------------
+
+
+def uuid_value(dialect_name: str) -> bytes | str:
+ """Return a UUID in the form the active dialect expects.
+
+ MySQL stores UUIDs as ``BINARY(16)`` (16 raw bytes); Postgres has a
+ native ``UUID`` type that accepts strings; SQLite stores them as
+ BLOB/TEXT and accepts either. Branching here keeps the seed script
+ backend-agnostic without depending on Superset's custom column types.
+ """
+ if dialect_name.startswith("mysql"):
+ return uuid4().bytes
+ return str(uuid4())
+
+
+@contextmanager
+def time_phase(name: str) -> Iterator[None]:
+ """Log elapsed wall time for a named phase."""
+ start = time.monotonic()
+ logger.info("[%s] starting", name)
+ try:
+ yield
+ finally:
+ elapsed = time.monotonic() - start
+ logger.info("[%s] done in %.2fs", name, elapsed)
+
+
+def count_rows(conn: Connection, table: str) -> int:
+ return conn.scalar(sa.text(f"SELECT COUNT(*) FROM {table}")) or 0 # noqa: S608
+
+
+def existing_ids(conn: Connection, table: str, limit: int | None = None) -> list[int]:
+ sql = f"SELECT id FROM {table} ORDER BY id" # noqa: S608
+ if limit is not None:
+ sql += f" LIMIT {limit}"
+ return [row[0] for row in conn.execute(sa.text(sql))]
+
+
+# ----------------------------------------------------------------------
+# Parent seeders
+#
+# Each function ensures the named parent table has at least ``target``
+# rows by inserting synthetic ones with minimal-but-valid columns.
+# Returns nothing; subsequent code reads back IDs via ``existing_ids``.
+# ----------------------------------------------------------------------
+
+
+def seed_dashboards(conn: Connection, target: int, dry_run: bool) -> None:
+ current = count_rows(conn, "dashboards")
+ if current >= target:
+ logger.info(
+ "dashboards: %d rows (target %d) — no insert needed", current, target
Review Comment:
**Suggestion:** This docstring says two UNIQUE junction tables are skipped, but the seeded set in this script only marks one (`dashboard_slices`) as UNIQUE. Correcting this avoids misleading operators about which seeded tables can receive duplicate rows during dirty-data runs. [comment mismatch]
<details>
<summary><b>Severity Level:</b> Minor 🧹</summary>
```mdx
- ⚠️ Docstring overgeneralizes which UNIQUE tables are skipped.
- ⚠️ Potential confusion when reasoning about dirty-data coverage.
- ⚠️ Suggestion affects comments only; no runtime misbehavior.
```
</details>
<details>
<summary><b>Steps of Reproduction ✅ </b></summary>
```mdx
1. In `scripts/seed_junction_load.py` around lines 176-182, inspect the docstring for
`_inject_dirty_data()`, which states: “Inject duplicate rows on every non-UNIQUE seeded
junction. The two tables that originally carried ``UNIQUE(fk1, fk2)`` are skipped because
their composite-PK successor (and their pre-migration UNIQUE constraint) both reject
duplicate inserts.”
2. At lines 86-93, review the `JUNCTIONS_WITH_UNIQUE` constant, which is the set used to
decide which seeded junction tables are skipped during dirty-data injection; it is defined
as `{"dashboard_slices"}` with an inline comment explaining that the other UNIQUE table
`report_schedule_user` is not included because this script does not seed it.
3. Inspect `_inject_dirty_data()`’s implementation at lines 176-193: it iterates `for
junction, fk1, fk2, _, _ in JUNCTIONS`, checks `if junction in JUNCTIONS_WITH_UNIQUE`, and
only skips those present in the set; because `JUNCTIONS` (lines 76-84) does not include
`report_schedule_user`, the only junction ever skipped in practice is `dashboard_slices`.
4. From these code paths, conclude that the docstring’s phrase “The two tables that
originally carried UNIQUE(fk1, fk2) are skipped” describes the broader migration context
but does not match the script’s seeded set, where only one such table is actually
considered; tightening this wording (for example, explicitly describing that only seeded
UNIQUE tables—currently just `dashboard_slices`—are skipped) would align the documentation
with the behavior implemented via `JUNCTIONS_WITH_UNIQUE`.
```
</details>
[](https://app.codeant.ai/fix-in-ide?tool=cursor&prompt_id=98bed46753ef40008f78d644671ef95d&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset) [](https://app.codeant.ai/fix-in-ide?tool=vscode-claude&prompt_id=98bed46753ef40008f78d644671ef95d&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset)
*(Use Cmd/Ctrl + Click for best experience)*
<details>
<summary><b>Prompt for AI Agent 🤖 </b></summary>
```mdx
This is a comment left during a code review.
**Path:** scripts/seed_junction_load.py
**Line:** 179:181
**Comment:**
*Comment Mismatch: This docstring says two UNIQUE junction tables are skipped, but the seeded set in this script only marks one (`dashboard_slices`) as UNIQUE. Correcting this avoids misleading operators about which seeded tables can receive duplicate rows during dirty-data runs.
Validate the correctness of the flagged issue. If correct, How can I resolve this? If you propose a fix, implement it and please make it concise.
Once fix is implemented, also check other comments on the same PR, and ask user if the user wants to fix the rest of the comments as well. if said yes, then fetch all the comments validate the correctness and implement a minimal fix
```
</details>
<a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F41076&comment_hash=9cde39adf3c323075a2bfdab123fd2c05df2d0940c5cd994db27938abff83de4&reaction=like'>👍</a> | <a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F41076&comment_hash=9cde39adf3c323075a2bfdab123fd2c05df2d0940c5cd994db27938abff83de4&reaction=dislike'>👎</a>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(versioning): cross-entity version activity view [superset]
Posted by "codeant-ai-for-open-source[bot] (via GitHub)" <gi...@apache.org>.
codeant-ai-for-open-source[bot] commented on code in PR #41076:
URL: https://github.com/apache/superset/pull/41076#discussion_r3500135902
##########
superset/dashboards/api.py:
##########
@@ -828,17 +887,32 @@ def put(self, pk: int) -> Response:
# This validates custom Schema with custom validations
except ValidationError as error:
return self.response_400(message=error.messages)
+
+ # Live version identifiers before the update (empty + query-free when
+ # ``ENABLE_VERSIONING_CAPTURE`` is off).
+ old_info = current_entity_version_info(Dashboard, pk)
+
try:
changed_model = UpdateDashboardCommand(pk, item).run()
last_modified_time = changed_model.changed_on.replace(
microsecond=0
).timestamp()
+ new_info = current_entity_version_info(
+ Dashboard, changed_model.id, changed_model.uuid
+ )
Review Comment:
**Suggestion:** Add an explicit type annotation for this newly introduced local variable to comply with the mandatory type-hint rule for relevant variables. [custom_rule]
**Severity Level:** Minor ⚠️
<details>
<summary><b>Why it matters? 🤔 </b></summary>
This newly added local variable is also unannotated even though it is assigned from a known helper return value. That is a real omission under the type-hint requirement.
</details>
[](https://app.codeant.ai/fix-in-ide?tool=cursor&prompt_id=55fd0d3edca94a1ebf3bf65a517f70d1&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset) [](https://app.codeant.ai/fix-in-ide?tool=vscode-claude&prompt_id=55fd0d3edca94a1ebf3bf65a517f70d1&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset)
*(Use Cmd/Ctrl + Click for best experience)*
<details>
<summary><b>Prompt for AI Agent 🤖 </b></summary>
```mdx
This is a comment left during a code review.
**Path:** superset/dashboards/api.py
**Line:** 900:902
**Comment:**
*Custom Rule: Add an explicit type annotation for this newly introduced local variable to comply with the mandatory type-hint rule for relevant variables.
Validate the correctness of the flagged issue. If correct, How can I resolve this? If you propose a fix, implement it and please make it concise.
Once fix is implemented, also check other comments on the same PR, and ask user if the user wants to fix the rest of the comments as well. if said yes, then fetch all the comments validate the correctness and implement a minimal fix
```
</details>
<a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F41076&comment_hash=a365287eaade1d1c29c5d8f460de8340ab35560fc212a65ca804970307c98f07&reaction=like'>👍</a> | <a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F41076&comment_hash=a365287eaade1d1c29c5d8f460de8340ab35560fc212a65ca804970307c98f07&reaction=dislike'>👎</a>
##########
superset/migrations/versions/2026-05-01_23-36_2bee73611e32_composite_pk_association_tables.py:
##########
@@ -0,0 +1,580 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""composite_pk_association_tables
+
+Replace the unused synthetic ``id INTEGER PRIMARY KEY`` on eight many-to-many
+association tables with a composite primary key on the two FK columns. Drops
+the now-redundant ``UniqueConstraint(fk1, fk2)`` on the two tables that
+already carry one. Pre-flight: deletes rows with NULL FK values (six tables
+allow them today) and any duplicate ``(fk1, fk2)`` rows.
+
+Motivated by SQLAlchemy-Continuum issue #129 (M2M restore against junction
+tables with surrogate PKs); also closes the data-integrity hole where six
+of the eight tables lacked DB-level uniqueness.
+
+Revision ID: 2bee73611e32
+Revises: 78a40c08b4be
+Create Date: 2026-05-01 23:36:34.050058
+
+"""
+
+import logging
+from typing import NamedTuple
+
+import sqlalchemy as sa
+from alembic import op
+from alembic.operations.base import BatchOperations
+from sqlalchemy import inspect
+from sqlalchemy.engine import Connection
+
+# revision identifiers, used by Alembic.
+revision = "2bee73611e32"
+down_revision = "78a40c08b4be"
Review Comment:
**Suggestion:** Add explicit type annotations for the Alembic revision metadata variables to satisfy the type-hint requirement for annotatable module-level variables. [custom_rule]
**Severity Level:** Minor ⚠️
<details>
<summary><b>Why it matters? 🤔 </b></summary>
These module-level variables are plain Python assignments and can be annotated, so they fit the type-hint rule for annotatable variables. The file already uses type hints elsewhere, but these Alembic metadata variables omit them.
</details>
[](https://app.codeant.ai/fix-in-ide?tool=cursor&prompt_id=c9eb0fc247444f80b6e717321f5b968d&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset) [](https://app.codeant.ai/fix-in-ide?tool=vscode-claude&prompt_id=c9eb0fc247444f80b6e717321f5b968d&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset)
*(Use Cmd/Ctrl + Click for best experience)*
<details>
<summary><b>Prompt for AI Agent 🤖 </b></summary>
```mdx
This is a comment left during a code review.
**Path:** superset/migrations/versions/2026-05-01_23-36_2bee73611e32_composite_pk_association_tables.py
**Line:** 45:46
**Comment:**
*Custom Rule: Add explicit type annotations for the Alembic revision metadata variables to satisfy the type-hint requirement for annotatable module-level variables.
Validate the correctness of the flagged issue. If correct, How can I resolve this? If you propose a fix, implement it and please make it concise.
Once fix is implemented, also check other comments on the same PR, and ask user if the user wants to fix the rest of the comments as well. if said yes, then fetch all the comments validate the correctness and implement a minimal fix
```
</details>
<a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F41076&comment_hash=3884dc7e703913a5741979baf634f529d41d14792b35e32d6c351552e8150680&reaction=like'>👍</a> | <a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F41076&comment_hash=3884dc7e703913a5741979baf634f529d41d14792b35e32d6c351552e8150680&reaction=dislike'>👎</a>
##########
superset/dashboards/api.py:
##########
@@ -828,17 +887,32 @@ def put(self, pk: int) -> Response:
# This validates custom Schema with custom validations
except ValidationError as error:
return self.response_400(message=error.messages)
+
+ # Live version identifiers before the update (empty + query-free when
+ # ``ENABLE_VERSIONING_CAPTURE`` is off).
+ old_info = current_entity_version_info(Dashboard, pk)
Review Comment:
**Suggestion:** Add an explicit variable type annotation for this new local value (for example, the shared version-info dataclass type) to satisfy the type-hint requirement. [custom_rule]
**Severity Level:** Minor ⚠️
<details>
<summary><b>Why it matters? 🤔 </b></summary>
The new local variable is introduced without any type annotation, and its return type is known/annotatable. This matches the python type-hint rule for relevant variables.
</details>
[](https://app.codeant.ai/fix-in-ide?tool=cursor&prompt_id=01aa8ba3281e43a98f2bf8814e2d1dbf&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset) [](https://app.codeant.ai/fix-in-ide?tool=vscode-claude&prompt_id=01aa8ba3281e43a98f2bf8814e2d1dbf&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset)
*(Use Cmd/Ctrl + Click for best experience)*
<details>
<summary><b>Prompt for AI Agent 🤖 </b></summary>
```mdx
This is a comment left during a code review.
**Path:** superset/dashboards/api.py
**Line:** 893:893
**Comment:**
*Custom Rule: Add an explicit variable type annotation for this new local value (for example, the shared version-info dataclass type) to satisfy the type-hint requirement.
Validate the correctness of the flagged issue. If correct, How can I resolve this? If you propose a fix, implement it and please make it concise.
Once fix is implemented, also check other comments on the same PR, and ask user if the user wants to fix the rest of the comments as well. if said yes, then fetch all the comments validate the correctness and implement a minimal fix
```
</details>
<a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F41076&comment_hash=379cbc5e2b26d3c8daa461664e09b6bb30b0c5d18e4ef28dd72681ceab9af52a&reaction=like'>👍</a> | <a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F41076&comment_hash=379cbc5e2b26d3c8daa461664e09b6bb30b0c5d18e4ef28dd72681ceab9af52a&reaction=dislike'>👎</a>
##########
superset/migrations/versions/2026-05-28_19-50_56cd24c07170_add_versioning_tables.py:
##########
@@ -0,0 +1,562 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""add_versioning_tables
+
+Creates the full schema backing entity versioning in a single
+migration:
+
+1. ``version_transaction`` — audit log keyed by Continuum's per-flush
+ transaction id (plus a Postgres-specific id sequence).
+2. **Parent shadow tables** mirroring each versioned entity's columns:
+ ``dashboards_version`` / ``slices_version`` / ``tables_version``.
+3. ``version_changes`` — field-level diff log keyed to a
+ ``(transaction, entity)`` pair; each row describes one atomic change
+ (one field or one child-collection element) that occurred during a
+ save.
+4. **Child shadow tables** for the collections Continuum auto-registers
+ when ``__versioned__`` is applied to ``TableColumn`` / ``SqlMetric``
+ and the ``slices`` exclude is removed from
+ ``Dashboard.__versioned__``: ``table_columns_version`` /
+ ``sql_metrics_version`` / ``dashboard_slices_version``.
+
+All shadow tables follow the validity-strategy shape (mirrored columns
++ ``transaction_id`` / ``end_transaction_id`` / ``operation_type``
+bookkeeping with FKs to ``version_transaction.id``). The current
+version row has ``end_transaction_id = NULL``.
+
+This migration replaces three iterative migrations from the spike phase
+(``56cd24c07170``, ``e1f3c5a7b9d0``, ``f7a2b3c4d5e6``) that captured the
+same schema in three steps as the feature was developed. Compacting
+gives downstream operators one migration to apply / reverse and one
+review surface. The ``revision`` hash is reused from the original first
+migration so anyone still tracking the chain by that hash lands on the
+same logical change set.
+
+Generated by hand because the current Continuum + Alembic-autogenerate
+interaction trips on the renamed ``transaction`` -> ``version_transaction``
+table key (``KeyError`` lookups in ``table_key_to_table``). Column
+inventories were sourced from the live model ``__table__`` definitions
+and ``version_class(...).__table__`` / Continuum association metadata.
+
+Primary key choice. Both ``version_transaction.id`` and
+``version_changes.id`` are ``BigInteger`` autoincrement — a deliberate
+carveout from the project's UUID-PK convention for new models (see
+``CLAUDE.md`` §"UUID Migration"). ``version_transaction`` is keyed
+externally by SQLAlchemy-Continuum via
+``nextval('version_transaction_id_seq')`` on every INSERT; matching
+that contract is required for ``versioning_manager`` to function.
+``version_changes`` follows the same shape because the user-facing
+identity is the ``(transaction_id, entity_kind, entity_id, sequence)``
+composite unique key, not the row id; the API surfaces a deterministic
+UUIDv5 ``version_uuid`` derived from ``entity.uuid`` and
+``transaction_id`` for stable external references.
+
+Revision ID: 56cd24c07170
+Revises: 2bee73611e32
+Create Date: 2026-05-28 19:50:00.000000
+
+"""
+
+from __future__ import annotations
+
+import sqlalchemy as sa
+from alembic import op
+from sqlalchemy_utils import UUIDType
+
+from superset.utils.core import MediumText
+
+revision = "56cd24c07170"
+# Stacked on the composite-PK association-tables change (2bee73611e32) so the
+# Continuum shadow tables this migration creates can mirror the
+# composite-PK shape of the live association tables. If that change
+# is removed from the stack, this should be reverted to "ce6bd21901ab".
+down_revision = "2bee73611e32"
+
+
+def upgrade() -> None:
+ bind = op.get_bind()
+
+ # ------------------------------------------------------------------
+ # version_transaction
+ #
+ # Audit log for each versioning event. Continuum emits
+ # ``nextval('version_transaction_id_seq')`` on every INSERT, so the
+ # sequence must exist before the table on Postgres. SQLite/MySQL
+ # ignore the explicit CREATE SEQUENCE (they auto-increment natively).
+ # ------------------------------------------------------------------
+ if bind.dialect.name == "postgresql":
+ op.execute("CREATE SEQUENCE IF NOT EXISTS version_transaction_id_seq")
+
+ op.create_table(
Review Comment:
**Suggestion:** Use the migration helper from `superset.migrations.shared.utils` for table creation instead of calling Alembic operations directly. [custom_rule]
**Severity Level:** Minor ⚠️
<details>
<summary><b>Why it matters? 🤔 </b></summary>
The shared migration helpers include a create_table wrapper in superset.migrations.shared.utils, so using bare op.create_table in a migration is exactly the kind of raw operation this rule flags.
</details>
[](https://app.codeant.ai/fix-in-ide?tool=cursor&prompt_id=8f6aca01d82e4c51a8e270355e794ad1&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset) [](https://app.codeant.ai/fix-in-ide?tool=vscode-claude&prompt_id=8f6aca01d82e4c51a8e270355e794ad1&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset)
*(Use Cmd/Ctrl + Click for best experience)*
<details>
<summary><b>Prompt for AI Agent 🤖 </b></summary>
```mdx
This is a comment left during a code review.
**Path:** superset/migrations/versions/2026-05-28_19-50_56cd24c07170_add_versioning_tables.py
**Line:** 104:104
**Comment:**
*Custom Rule: Use the migration helper from `superset.migrations.shared.utils` for table creation instead of calling Alembic operations directly.
Validate the correctness of the flagged issue. If correct, How can I resolve this? If you propose a fix, implement it and please make it concise.
Once fix is implemented, also check other comments on the same PR, and ask user if the user wants to fix the rest of the comments as well. if said yes, then fetch all the comments validate the correctness and implement a minimal fix
```
</details>
<a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F41076&comment_hash=25163d79cf0ecfb53c481d6854e15ed368a0f6d3fb04410912b50b6a63d55e6f&reaction=like'>👍</a> | <a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F41076&comment_hash=25163d79cf0ecfb53c481d6854e15ed368a0f6d3fb04410912b50b6a63d55e6f&reaction=dislike'>👎</a>
##########
superset/migrations/versions/2026-05-01_23-36_2bee73611e32_composite_pk_association_tables.py:
##########
@@ -0,0 +1,580 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""composite_pk_association_tables
+
+Replace the unused synthetic ``id INTEGER PRIMARY KEY`` on eight many-to-many
+association tables with a composite primary key on the two FK columns. Drops
+the now-redundant ``UniqueConstraint(fk1, fk2)`` on the two tables that
+already carry one. Pre-flight: deletes rows with NULL FK values (six tables
+allow them today) and any duplicate ``(fk1, fk2)`` rows.
+
+Motivated by SQLAlchemy-Continuum issue #129 (M2M restore against junction
+tables with surrogate PKs); also closes the data-integrity hole where six
+of the eight tables lacked DB-level uniqueness.
+
+Revision ID: 2bee73611e32
+Revises: 78a40c08b4be
+Create Date: 2026-05-01 23:36:34.050058
+
+"""
+
+import logging
+from typing import NamedTuple
+
+import sqlalchemy as sa
+from alembic import op
+from alembic.operations.base import BatchOperations
+from sqlalchemy import inspect
+from sqlalchemy.engine import Connection
+
+# revision identifiers, used by Alembic.
+revision = "2bee73611e32"
+down_revision = "78a40c08b4be"
+
+logger = logging.getLogger("alembic.env")
+
+
+class AssociationTable(NamedTuple):
+ """A junction table being converted from surrogate-id PK to composite-FK PK."""
+
+ name: str
+ fk1: str
+ fk2: str
+
+
+# Order is alphabetical by table name; deterministic for review and bisection.
+AFFECTED_TABLES: list[AssociationTable] = [
+ AssociationTable("dashboard_roles", "dashboard_id", "role_id"),
+ AssociationTable("dashboard_slices", "dashboard_id", "slice_id"),
+ AssociationTable("dashboard_user", "user_id", "dashboard_id"),
+ AssociationTable("report_schedule_user", "user_id", "report_schedule_id"),
+ AssociationTable("rls_filter_roles", "role_id", "rls_filter_id"),
+ AssociationTable("rls_filter_tables", "table_id", "rls_filter_id"),
+ AssociationTable("slice_user", "user_id", "slice_id"),
+ AssociationTable("sqlatable_user", "user_id", "table_id"),
+]
+
+# These two tables already declare ``UniqueConstraint(fk1, fk2)`` in the model;
+# the composite PK subsumes it, so the migration drops the redundant constraint.
+TABLES_WITH_PRE_EXISTING_UNIQUE: set[str] = {
+ "dashboard_slices",
+ "report_schedule_user",
+}
+
+# Documentation set: tables whose FK columns are nullable in their original
+# create_table migrations (``dashboard_roles.dashboard_id`` from revision
+# e11ccdd12658 is the most recent addition). ``report_schedule_user`` is the
+# only affected table created with both FK columns ``NOT NULL`` and is
+# intentionally absent here. This set is no longer consulted at runtime — the
+# upgrade now runs the NULL-FK cleanup on every affected table because the
+# DELETE is a cheap no-op when the columns are already NOT NULL, and that
+# eliminates the risk of bugs from this set going stale (the
+# ``dashboard_roles`` omission caught in PR review was exactly that bug).
+TABLES_WITH_NULLABLE_FKS: set[str] = {
+ "dashboard_roles",
+ "dashboard_slices",
+ "dashboard_user",
+ "rls_filter_roles",
+ "rls_filter_tables",
+ "slice_user",
+ "sqlatable_user",
+}
+
+
+def _check_no_external_fks_to_id(conn: Connection) -> None:
+ """Raise ``RuntimeError`` if any foreign key in the database references one
+ of the eight junction-table ``id`` columns. Uses SQLAlchemy's ``Inspector``
+ for dialect-agnostic introspection across PostgreSQL, MySQL, and SQLite.
+
+ Scope limitation: ``Inspector.get_table_names()`` returns tables in the
+ connection's default schema only. On PostgreSQL deployments where Superset
+ metadata lives in a non-default schema, or on multi-schema deployments
+ that allow cross-schema FKs, an external FK in another schema would not
+ be detected. This is acceptable for the standard single-schema
+ deployment that Superset documents; operators with multi-schema
+ metadata should run the equivalent inventory query against
+ ``information_schema.referential_constraints`` themselves before
+ applying.
+ """
+ affected = {t.name for t in AFFECTED_TABLES}
+ insp = inspect(conn)
+ for table_name in insp.get_table_names():
+ if table_name in affected:
+ continue
+ for fk in insp.get_foreign_keys(table_name):
+ if fk["referred_table"] in affected and "id" in fk["referred_columns"]:
+ raise RuntimeError(
+ f"Cannot drop synthetic id from {fk['referred_table']}: "
+ f"external FK {fk.get('name', '<unnamed>')} on {table_name} "
+ f"references {fk['referred_table']}({fk['referred_columns']}). "
+ "Drop or migrate the referencing FK before applying this "
+ "migration."
+ )
+
+
+def _table_clause(t: AssociationTable) -> sa.sql.expression.TableClause:
+ """Build a lightweight SQLAlchemy ``TableClause`` for ``t`` exposing the
+ columns the helper queries reference (``id``, ``fk1``, ``fk2``). Used so
+ that the dedupe / cleanup / assert SQL can be expressed via SQLAlchemy
+ core constructs rather than via string interpolation."""
+ return sa.table(t.name, sa.column("id"), sa.column(t.fk1), sa.column(t.fk2))
+
+
+def _delete_null_fk_rows(conn: Connection, t: AssociationTable) -> int:
+ """Delete rows where ``t.fk1`` or ``t.fk2`` is NULL on ``t.name``.
+
+ Returns the deletion count. Required because primary-key columns must be
+ NOT NULL; the PK-add downstream would fail with a cryptic constraint
+ violation if any NULL-FK rows survived. Run unconditionally on every
+ affected table — see ``TABLES_WITH_NULLABLE_FKS`` above for the rationale.
+ """
+ tbl = _table_clause(t)
+ stmt = sa.delete(tbl).where(sa.or_(tbl.c[t.fk1].is_(None), tbl.c[t.fk2].is_(None)))
+ result = conn.execute(stmt)
+ n = result.rowcount or 0
+ if n:
+ logger.warning(
+ "Deleted %d row(s) with NULL FK from %s before composite-PK promotion",
+ n,
+ t.name,
+ )
+ return n
+
+
+def _dedupe_by_min_id(conn: Connection, t: AssociationTable) -> int:
+ """Delete duplicate ``(t.fk1, t.fk2)`` rows from ``t.name`` keeping ``MIN(id)``.
+
+ Returns the deletion count. The ``NOT IN`` argument is wrapped in an
+ extra ``SELECT keep_id FROM (...) AS s`` derived table because MySQL
+ rejects ``DELETE FROM t WHERE id NOT IN (SELECT MIN(id) FROM t GROUP BY
+ ...)`` with ERROR 1093 unless the inner SELECT is materialized through
+ a derived table. SQLAlchemy's ``.subquery()`` produces that wrap.
+
+ Logs a sample (up to 10) of the discarded ``(fk1, fk2, id)`` tuples at
+ WARN before deletion, so operators can audit which rows are dropped —
+ the "keep ``MIN(id)``" policy preserves the original row, which is
+ correct in practice but discards any later, semantically-identical
+ re-grants.
+ """
+ tbl = _table_clause(t)
+
+ keep_min = (
+ sa.select(sa.func.min(tbl.c.id).label("keep_id"))
+ .group_by(tbl.c[t.fk1], tbl.c[t.fk2])
+ .subquery("keep_min")
+ )
+ keep_ids = sa.select(keep_min.c.keep_id)
+ discarded = tbl.c.id.notin_(keep_ids)
+
+ sample_stmt = (
+ sa.select(tbl.c[t.fk1], tbl.c[t.fk2], tbl.c.id).where(discarded).limit(10)
+ )
+ sample = list(conn.execute(sample_stmt))
+
+ delete_stmt = sa.delete(tbl).where(discarded)
+ result = conn.execute(delete_stmt)
+ n = result.rowcount or 0
+ if n:
+ logger.warning(
+ "Deduped %d duplicate row(s) from %s; sample of discarded "
+ "(%s, %s, id) tuples (up to 10): %s",
+ n,
+ t.name,
+ t.fk1,
+ t.fk2,
+ sample,
+ )
+ return n
+
+
+def _assert_no_duplicates(conn: Connection, t: AssociationTable) -> None:
+ """Raise ``RuntimeError`` if any ``(t.fk1, t.fk2)`` duplicate group remains.
+
+ Called after ``_dedupe_by_min_id`` to surface silent dialect-dependent
+ dedupe failures (e.g., a MySQL syntax issue) as an actionable error
+ before the PK-add fires with a less-helpful constraint-violation message.
+ """
+ tbl = _table_clause(t)
+ duplicate_groups = (
+ sa.select(sa.literal(1))
+ .select_from(tbl)
+ .group_by(tbl.c[t.fk1], tbl.c[t.fk2])
+ .having(sa.func.count() > 1)
+ .subquery("duplicate_groups")
+ )
+ count_stmt = sa.select(sa.func.count()).select_from(duplicate_groups)
+ if remaining := conn.scalar(count_stmt) or 0:
+ raise RuntimeError(
+ f"Dedupe failed for {t.name}: {remaining} duplicate "
+ f"({t.fk1}, {t.fk2}) groups remain after _dedupe_by_min_id. "
+ f"Check the dedupe SQL for dialect {conn.dialect.name}."
+ )
+
+
+def _build_pre_upgrade_table(
+ insp: sa.engine.reflection.Inspector,
+ t: AssociationTable,
+ fks: list[dict] | None = None,
+) -> sa.Table:
+ """Build a ``Table`` object representing the pre-upgrade schema of ``t``,
+ explicitly *without* any redundant ``UniqueConstraint(t.fk1, t.fk2)``.
+ Used as ``copy_from`` to ``batch_alter_table`` so the rebuilt table
+ omits the unnamed UNIQUE constraint deterministically across dialects
+ (SQLite reflects unnamed UNIQUEs with ``name=None``, defeating the
+ standard ``batch_op.drop_constraint(name)`` path).
+
+ Reflects column types and FK targets (with original FK constraint names
+ preserved) from the live database; only the redundant UNIQUE is omitted.
+
+ *fks* lets a caller pass a pre-captured ``get_foreign_keys`` result.
+ The MySQL upgrade path drops the live FK constraints before building
+ this table, so re-reflecting here would only see them via the
+ Inspector's per-instance ``info_cache`` — an implementation detail,
+ not a contract. Passing the pre-drop list makes the dependency
+ explicit instead of relying on reflection caching.
+ """
+ md = sa.MetaData()
+ if fks is None:
+ fks = insp.get_foreign_keys(t.name)
+ fks_for_col: dict[str, list[dict]] = {}
+ for fk in fks:
+ for col_name in fk["constrained_columns"]:
+ fks_for_col.setdefault(col_name, []).append(fk)
+
+ cols: list[sa.Column] = []
+ for c in insp.get_columns(t.name):
+ col_kwargs = {"nullable": c.get("nullable", True)}
+ if c["name"] == "id":
+ col_kwargs["primary_key"] = True
+ col_kwargs["autoincrement"] = True
+ fk_args = []
+ for fk in fks_for_col.get(c["name"], []):
+ idx = fk["constrained_columns"].index(c["name"])
+ target = f"{fk['referred_table']}.{fk['referred_columns'][idx]}"
+ options = {}
+ if fk.get("options", {}).get("ondelete"):
+ options["ondelete"] = fk["options"]["ondelete"]
+ if fk.get("name"):
+ options["name"] = fk["name"]
+ fk_args.append(sa.ForeignKey(target, **options))
+ cols.append(sa.Column(c["name"], c["type"], *fk_args, **col_kwargs))
+ return sa.Table(t.name, md, *cols)
+
+
+def _drop_redundant_unique_by_name(
+ conn: Connection, insp: sa.engine.reflection.Inspector, t: AssociationTable
+) -> None:
+ """Drop the redundant ``UNIQUE(fk1, fk2)`` constraint by its reflected
+ name on PostgreSQL / MySQL.
+
+ The two tables in ``TABLES_WITH_PRE_EXISTING_UNIQUE`` carry a UNIQUE
+ constraint that the composite primary key subsumes. PostgreSQL and
+ MySQL both auto-name UNIQUE constraints (``<table>_<cols>_key`` on
+ Postgres, ``<table>_<col>_<n>`` or the explicit ``uq_*`` we may have
+ given it on MySQL), so they're reflectable by name. SQLite is
+ handled separately via ``recreate="always"`` + ``copy_from`` because
+ it reflects unnamed UNIQUEs with ``name=None``.
+
+ No-op if no matching UNIQUE is found (defensive — re-runs after a
+ partial application should not error).
+ """
+ for uc in insp.get_unique_constraints(t.name):
+ if set(uc.get("column_names", [])) == {t.fk1, t.fk2} and uc.get("name"):
+ op.drop_constraint(uc["name"], t.name, type_="unique")
+ return
+
+
+# MySQL ON DELETE actions that the downgrade re-create loop is allowed
+# to interpolate into raw SQL. The reflected value comes from MySQL's
+# information_schema (so not user input), but a whitelist eliminates
+# the "what if an unexpected value appears" question entirely. The
+# four entries are the SQL-standard set; SET DEFAULT is intentionally
+# excluded because InnoDB silently downgrades it to NO ACTION.
+_VALID_ONDELETE_ACTIONS: frozenset[str] = frozenset(
+ {"CASCADE", "SET NULL", "RESTRICT", "NO ACTION"}
+)
+
+
+def _enforce_not_null_for_sqlite(
+ batch_op: BatchOperations, t: AssociationTable, conn: Connection
+) -> None:
+ """Force ``NOT NULL`` on the FK columns post-PK-promotion on SQLite only.
+
+ SQLite has a long-standing quirk: composite ``PRIMARY KEY`` does not
+ promote constituent columns to ``NOT NULL`` (only ``INTEGER PRIMARY KEY``
+ does). PostgreSQL and MySQL implicitly promote the PK columns to
+ ``NOT NULL`` when the constraint is added, making the explicit
+ ``alter_column`` redundant there.
+
+ Skipping the ``alter_column`` on MySQL is also functionally required:
+ MySQL 8 rejects ``ALTER COLUMN`` on a column that participates in a
+ foreign key constraint with ``ERROR 1832 (HY000): Cannot change column
+ 'X': used in a foreign key constraint 'Y'`` whenever the table has
+ data — even when the only change is ``NULL`` → ``NOT NULL`` and the
+ column is already part of a freshly-added composite primary key (which
+ InnoDB has just made implicitly ``NOT NULL`` anyway). The error fires
+ on populated tables but not on empty ones, which is why CI's
+ ``test-mysql`` shard (fresh schema) didn't catch this and a real
+ production-shaped install does.
+
+ Only SQLite still needs the explicit step, and SQLite has no FK
+ enforcement objection.
+ """
+ if conn.dialect.name == "sqlite":
+ batch_op.alter_column(t.fk1, existing_type=sa.Integer, nullable=False)
+ batch_op.alter_column(t.fk2, existing_type=sa.Integer, nullable=False)
+
+
+def upgrade() -> None:
+ conn = op.get_bind()
+ _check_no_external_fks_to_id(conn)
+ insp = inspect(conn)
+
+ for t in AFFECTED_TABLES:
+ # Resumability guard: on MySQL every DDL statement auto-commits, so
+ # a failure at table N of 8 leaves tables 1..N-1 already converted
+ # while ``alembic_version`` is still un-stamped. Without this guard
+ # a re-run would fail at table 1 (``drop_column("id")`` on a table
+ # that no longer has ``id``), and ``downgrade`` can't run either
+ # (the revision was never stamped) — recovery would need manual
+ # surgery. A converted table is identified by the absent ``id``
+ # column; skipping it makes re-running the upgrade safe on every
+ # dialect (Postgres/SQLite wrap the migration in a transaction, so
+ # the guard is simply never hit there).
+ if "id" not in {c["name"] for c in insp.get_columns(t.name)}:
+ logger.info(
+ "%s: already converted (no surrogate id column); skipping",
+ t.name,
+ )
+ continue
+
+ # Run NULL-FK cleanup unconditionally: it is a no-op DELETE on tables
+ # whose FK columns are already NOT NULL (cheap), and skipping it on a
+ # table whose FK was nullable would leave the PK-add to fail with a
+ # cryptic constraint violation. Cf. ``TABLES_WITH_NULLABLE_FKS`` above
+ # for documentation of which tables are known to have nullable FKs.
+ _delete_null_fk_rows(conn, t)
+ _dedupe_by_min_id(conn, t)
+ _assert_no_duplicates(conn, t)
+
+ # Two tables (``dashboard_slices``, ``report_schedule_user``)
+ # carry a redundant ``UNIQUE(fk1, fk2)`` that the composite PK
+ # subsumes. Three dialect-specific paths:
+ #
+ # * **PostgreSQL** — the UNIQUE constraint has a stable
+ # reflected name (Postgres default convention), so we
+ # ``DROP CONSTRAINT`` by name and then run the structural
+ # change as direct ALTER. This avoids the full-table copy
+ # that ``recreate="always"`` would trigger
+ # (``CREATE TABLE AS SELECT → DROP → RENAME``), holding
+ # ``ACCESS EXCLUSIVE`` only for the (much shorter) PK
+ # index build instead of the full copy duration.
+ #
+ # * **MySQL** — InnoDB binds the FK constraints to the
+ # redundant UNIQUE's underlying index for back-reference,
+ # so a direct ``DROP CONSTRAINT`` of the UNIQUE raises
+ # ``ERROR 1553``. Use ``recreate="always"`` to rebuild the
+ # table without the UNIQUE; drop the FKs first to dodge
+ # the ``ERROR 1826`` (duplicate FK constraint name) that
+ # the temp-table phase would otherwise provoke. The FKs
+ # are re-created automatically as part of ``copy_from``.
+ #
+ # * **SQLite** — unnamed UNIQUE constraints reflect with
+ # ``name=None`` and can't be dropped by name. Use
+ # ``recreate="always"`` + ``copy_from`` (omits UNIQUE).
+ # SQLite always rebuilds for PK changes anyway, so the
+ # recreate isn't extra cost there.
+ if t.name in TABLES_WITH_PRE_EXISTING_UNIQUE:
+ if conn.dialect.name == "postgresql":
+ _drop_redundant_unique_by_name(conn, insp, t)
+ with op.batch_alter_table(t.name) as batch_op:
+ batch_op.drop_column("id")
+ batch_op.create_primary_key(f"pk_{t.name}", [t.fk1, t.fk2])
+ _enforce_not_null_for_sqlite(batch_op, t, conn)
+ else:
+ # Capture the FK list BEFORE dropping: the copy_from table
+ # below must embed these constraints, and re-reflecting
+ # after the drop only works via the Inspector's
+ # per-instance info_cache (see _build_pre_upgrade_table).
+ pre_drop_fks = insp.get_foreign_keys(t.name)
+ if conn.dialect.name == "mysql":
+ for fk in pre_drop_fks:
+ if fk_name := fk.get("name"):
+ op.drop_constraint(fk_name, t.name, type_="foreignkey")
Review Comment:
**Suggestion:** Use the foreign-key helper utilities from `superset.migrations.shared.utils` instead of manually dropping constraints with low-level Alembic operations. [custom_rule]
**Severity Level:** Minor ⚠️
<details>
<summary><b>Why it matters? 🤔 </b></summary>
The migration shared utilities include `drop_fks_for_table`, which performs dialect-aware FK dropping. This code manually drops foreign keys with Alembic operations instead of using the shared helper, so it matches the custom migration-utils rule.
</details>
[](https://app.codeant.ai/fix-in-ide?tool=cursor&prompt_id=a0bb4b1f296e48eb8d0d45be74789996&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset) [](https://app.codeant.ai/fix-in-ide?tool=vscode-claude&prompt_id=a0bb4b1f296e48eb8d0d45be74789996&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset)
*(Use Cmd/Ctrl + Click for best experience)*
<details>
<summary><b>Prompt for AI Agent 🤖 </b></summary>
```mdx
This is a comment left during a code review.
**Path:** superset/migrations/versions/2026-05-01_23-36_2bee73611e32_composite_pk_association_tables.py
**Line:** 415:417
**Comment:**
*Custom Rule: Use the foreign-key helper utilities from `superset.migrations.shared.utils` instead of manually dropping constraints with low-level Alembic operations.
Validate the correctness of the flagged issue. If correct, How can I resolve this? If you propose a fix, implement it and please make it concise.
Once fix is implemented, also check other comments on the same PR, and ask user if the user wants to fix the rest of the comments as well. if said yes, then fetch all the comments validate the correctness and implement a minimal fix
```
</details>
<a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F41076&comment_hash=737f9f7b664a26c8a75ba417d9591bffb3581a1d66ca1331ad03ee7cb560bb1e&reaction=like'>👍</a> | <a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F41076&comment_hash=737f9f7b664a26c8a75ba417d9591bffb3581a1d66ca1331ad03ee7cb560bb1e&reaction=dislike'>👎</a>
##########
superset/migrations/versions/2026-05-01_23-36_2bee73611e32_composite_pk_association_tables.py:
##########
@@ -0,0 +1,580 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""composite_pk_association_tables
+
+Replace the unused synthetic ``id INTEGER PRIMARY KEY`` on eight many-to-many
+association tables with a composite primary key on the two FK columns. Drops
+the now-redundant ``UniqueConstraint(fk1, fk2)`` on the two tables that
+already carry one. Pre-flight: deletes rows with NULL FK values (six tables
+allow them today) and any duplicate ``(fk1, fk2)`` rows.
+
+Motivated by SQLAlchemy-Continuum issue #129 (M2M restore against junction
+tables with surrogate PKs); also closes the data-integrity hole where six
+of the eight tables lacked DB-level uniqueness.
+
+Revision ID: 2bee73611e32
+Revises: 78a40c08b4be
+Create Date: 2026-05-01 23:36:34.050058
+
+"""
+
+import logging
+from typing import NamedTuple
+
+import sqlalchemy as sa
+from alembic import op
+from alembic.operations.base import BatchOperations
+from sqlalchemy import inspect
+from sqlalchemy.engine import Connection
+
+# revision identifiers, used by Alembic.
+revision = "2bee73611e32"
+down_revision = "78a40c08b4be"
+
+logger = logging.getLogger("alembic.env")
+
+
+class AssociationTable(NamedTuple):
+ """A junction table being converted from surrogate-id PK to composite-FK PK."""
+
+ name: str
+ fk1: str
+ fk2: str
+
+
+# Order is alphabetical by table name; deterministic for review and bisection.
+AFFECTED_TABLES: list[AssociationTable] = [
+ AssociationTable("dashboard_roles", "dashboard_id", "role_id"),
+ AssociationTable("dashboard_slices", "dashboard_id", "slice_id"),
+ AssociationTable("dashboard_user", "user_id", "dashboard_id"),
+ AssociationTable("report_schedule_user", "user_id", "report_schedule_id"),
+ AssociationTable("rls_filter_roles", "role_id", "rls_filter_id"),
+ AssociationTable("rls_filter_tables", "table_id", "rls_filter_id"),
+ AssociationTable("slice_user", "user_id", "slice_id"),
+ AssociationTable("sqlatable_user", "user_id", "table_id"),
+]
+
+# These two tables already declare ``UniqueConstraint(fk1, fk2)`` in the model;
+# the composite PK subsumes it, so the migration drops the redundant constraint.
+TABLES_WITH_PRE_EXISTING_UNIQUE: set[str] = {
+ "dashboard_slices",
+ "report_schedule_user",
+}
+
+# Documentation set: tables whose FK columns are nullable in their original
+# create_table migrations (``dashboard_roles.dashboard_id`` from revision
+# e11ccdd12658 is the most recent addition). ``report_schedule_user`` is the
+# only affected table created with both FK columns ``NOT NULL`` and is
+# intentionally absent here. This set is no longer consulted at runtime — the
+# upgrade now runs the NULL-FK cleanup on every affected table because the
+# DELETE is a cheap no-op when the columns are already NOT NULL, and that
+# eliminates the risk of bugs from this set going stale (the
+# ``dashboard_roles`` omission caught in PR review was exactly that bug).
+TABLES_WITH_NULLABLE_FKS: set[str] = {
+ "dashboard_roles",
+ "dashboard_slices",
+ "dashboard_user",
+ "rls_filter_roles",
+ "rls_filter_tables",
+ "slice_user",
+ "sqlatable_user",
+}
+
+
+def _check_no_external_fks_to_id(conn: Connection) -> None:
+ """Raise ``RuntimeError`` if any foreign key in the database references one
+ of the eight junction-table ``id`` columns. Uses SQLAlchemy's ``Inspector``
+ for dialect-agnostic introspection across PostgreSQL, MySQL, and SQLite.
+
+ Scope limitation: ``Inspector.get_table_names()`` returns tables in the
+ connection's default schema only. On PostgreSQL deployments where Superset
+ metadata lives in a non-default schema, or on multi-schema deployments
+ that allow cross-schema FKs, an external FK in another schema would not
+ be detected. This is acceptable for the standard single-schema
+ deployment that Superset documents; operators with multi-schema
+ metadata should run the equivalent inventory query against
+ ``information_schema.referential_constraints`` themselves before
+ applying.
+ """
+ affected = {t.name for t in AFFECTED_TABLES}
+ insp = inspect(conn)
+ for table_name in insp.get_table_names():
+ if table_name in affected:
+ continue
+ for fk in insp.get_foreign_keys(table_name):
+ if fk["referred_table"] in affected and "id" in fk["referred_columns"]:
+ raise RuntimeError(
+ f"Cannot drop synthetic id from {fk['referred_table']}: "
+ f"external FK {fk.get('name', '<unnamed>')} on {table_name} "
+ f"references {fk['referred_table']}({fk['referred_columns']}). "
+ "Drop or migrate the referencing FK before applying this "
+ "migration."
+ )
+
+
+def _table_clause(t: AssociationTable) -> sa.sql.expression.TableClause:
+ """Build a lightweight SQLAlchemy ``TableClause`` for ``t`` exposing the
+ columns the helper queries reference (``id``, ``fk1``, ``fk2``). Used so
+ that the dedupe / cleanup / assert SQL can be expressed via SQLAlchemy
+ core constructs rather than via string interpolation."""
+ return sa.table(t.name, sa.column("id"), sa.column(t.fk1), sa.column(t.fk2))
+
+
+def _delete_null_fk_rows(conn: Connection, t: AssociationTable) -> int:
+ """Delete rows where ``t.fk1`` or ``t.fk2`` is NULL on ``t.name``.
+
+ Returns the deletion count. Required because primary-key columns must be
+ NOT NULL; the PK-add downstream would fail with a cryptic constraint
+ violation if any NULL-FK rows survived. Run unconditionally on every
+ affected table — see ``TABLES_WITH_NULLABLE_FKS`` above for the rationale.
+ """
+ tbl = _table_clause(t)
+ stmt = sa.delete(tbl).where(sa.or_(tbl.c[t.fk1].is_(None), tbl.c[t.fk2].is_(None)))
+ result = conn.execute(stmt)
+ n = result.rowcount or 0
+ if n:
+ logger.warning(
+ "Deleted %d row(s) with NULL FK from %s before composite-PK promotion",
+ n,
+ t.name,
+ )
+ return n
+
+
+def _dedupe_by_min_id(conn: Connection, t: AssociationTable) -> int:
+ """Delete duplicate ``(t.fk1, t.fk2)`` rows from ``t.name`` keeping ``MIN(id)``.
+
+ Returns the deletion count. The ``NOT IN`` argument is wrapped in an
+ extra ``SELECT keep_id FROM (...) AS s`` derived table because MySQL
+ rejects ``DELETE FROM t WHERE id NOT IN (SELECT MIN(id) FROM t GROUP BY
+ ...)`` with ERROR 1093 unless the inner SELECT is materialized through
+ a derived table. SQLAlchemy's ``.subquery()`` produces that wrap.
+
+ Logs a sample (up to 10) of the discarded ``(fk1, fk2, id)`` tuples at
+ WARN before deletion, so operators can audit which rows are dropped —
+ the "keep ``MIN(id)``" policy preserves the original row, which is
+ correct in practice but discards any later, semantically-identical
+ re-grants.
+ """
+ tbl = _table_clause(t)
+
+ keep_min = (
+ sa.select(sa.func.min(tbl.c.id).label("keep_id"))
+ .group_by(tbl.c[t.fk1], tbl.c[t.fk2])
+ .subquery("keep_min")
+ )
+ keep_ids = sa.select(keep_min.c.keep_id)
+ discarded = tbl.c.id.notin_(keep_ids)
+
+ sample_stmt = (
+ sa.select(tbl.c[t.fk1], tbl.c[t.fk2], tbl.c.id).where(discarded).limit(10)
+ )
+ sample = list(conn.execute(sample_stmt))
+
+ delete_stmt = sa.delete(tbl).where(discarded)
+ result = conn.execute(delete_stmt)
+ n = result.rowcount or 0
+ if n:
+ logger.warning(
+ "Deduped %d duplicate row(s) from %s; sample of discarded "
+ "(%s, %s, id) tuples (up to 10): %s",
+ n,
+ t.name,
+ t.fk1,
+ t.fk2,
+ sample,
+ )
+ return n
+
+
+def _assert_no_duplicates(conn: Connection, t: AssociationTable) -> None:
+ """Raise ``RuntimeError`` if any ``(t.fk1, t.fk2)`` duplicate group remains.
+
+ Called after ``_dedupe_by_min_id`` to surface silent dialect-dependent
+ dedupe failures (e.g., a MySQL syntax issue) as an actionable error
+ before the PK-add fires with a less-helpful constraint-violation message.
+ """
+ tbl = _table_clause(t)
+ duplicate_groups = (
+ sa.select(sa.literal(1))
+ .select_from(tbl)
+ .group_by(tbl.c[t.fk1], tbl.c[t.fk2])
+ .having(sa.func.count() > 1)
+ .subquery("duplicate_groups")
+ )
+ count_stmt = sa.select(sa.func.count()).select_from(duplicate_groups)
+ if remaining := conn.scalar(count_stmt) or 0:
+ raise RuntimeError(
+ f"Dedupe failed for {t.name}: {remaining} duplicate "
+ f"({t.fk1}, {t.fk2}) groups remain after _dedupe_by_min_id. "
+ f"Check the dedupe SQL for dialect {conn.dialect.name}."
+ )
+
+
+def _build_pre_upgrade_table(
+ insp: sa.engine.reflection.Inspector,
+ t: AssociationTable,
+ fks: list[dict] | None = None,
+) -> sa.Table:
+ """Build a ``Table`` object representing the pre-upgrade schema of ``t``,
+ explicitly *without* any redundant ``UniqueConstraint(t.fk1, t.fk2)``.
+ Used as ``copy_from`` to ``batch_alter_table`` so the rebuilt table
+ omits the unnamed UNIQUE constraint deterministically across dialects
+ (SQLite reflects unnamed UNIQUEs with ``name=None``, defeating the
+ standard ``batch_op.drop_constraint(name)`` path).
+
+ Reflects column types and FK targets (with original FK constraint names
+ preserved) from the live database; only the redundant UNIQUE is omitted.
+
+ *fks* lets a caller pass a pre-captured ``get_foreign_keys`` result.
+ The MySQL upgrade path drops the live FK constraints before building
+ this table, so re-reflecting here would only see them via the
+ Inspector's per-instance ``info_cache`` — an implementation detail,
+ not a contract. Passing the pre-drop list makes the dependency
+ explicit instead of relying on reflection caching.
+ """
+ md = sa.MetaData()
+ if fks is None:
+ fks = insp.get_foreign_keys(t.name)
+ fks_for_col: dict[str, list[dict]] = {}
+ for fk in fks:
+ for col_name in fk["constrained_columns"]:
+ fks_for_col.setdefault(col_name, []).append(fk)
+
+ cols: list[sa.Column] = []
+ for c in insp.get_columns(t.name):
+ col_kwargs = {"nullable": c.get("nullable", True)}
+ if c["name"] == "id":
+ col_kwargs["primary_key"] = True
+ col_kwargs["autoincrement"] = True
+ fk_args = []
+ for fk in fks_for_col.get(c["name"], []):
+ idx = fk["constrained_columns"].index(c["name"])
+ target = f"{fk['referred_table']}.{fk['referred_columns'][idx]}"
+ options = {}
+ if fk.get("options", {}).get("ondelete"):
+ options["ondelete"] = fk["options"]["ondelete"]
+ if fk.get("name"):
+ options["name"] = fk["name"]
+ fk_args.append(sa.ForeignKey(target, **options))
+ cols.append(sa.Column(c["name"], c["type"], *fk_args, **col_kwargs))
+ return sa.Table(t.name, md, *cols)
+
+
+def _drop_redundant_unique_by_name(
+ conn: Connection, insp: sa.engine.reflection.Inspector, t: AssociationTable
+) -> None:
+ """Drop the redundant ``UNIQUE(fk1, fk2)`` constraint by its reflected
+ name on PostgreSQL / MySQL.
+
+ The two tables in ``TABLES_WITH_PRE_EXISTING_UNIQUE`` carry a UNIQUE
+ constraint that the composite primary key subsumes. PostgreSQL and
+ MySQL both auto-name UNIQUE constraints (``<table>_<cols>_key`` on
+ Postgres, ``<table>_<col>_<n>`` or the explicit ``uq_*`` we may have
+ given it on MySQL), so they're reflectable by name. SQLite is
+ handled separately via ``recreate="always"`` + ``copy_from`` because
+ it reflects unnamed UNIQUEs with ``name=None``.
+
+ No-op if no matching UNIQUE is found (defensive — re-runs after a
+ partial application should not error).
+ """
+ for uc in insp.get_unique_constraints(t.name):
+ if set(uc.get("column_names", [])) == {t.fk1, t.fk2} and uc.get("name"):
+ op.drop_constraint(uc["name"], t.name, type_="unique")
+ return
+
+
+# MySQL ON DELETE actions that the downgrade re-create loop is allowed
+# to interpolate into raw SQL. The reflected value comes from MySQL's
+# information_schema (so not user input), but a whitelist eliminates
+# the "what if an unexpected value appears" question entirely. The
+# four entries are the SQL-standard set; SET DEFAULT is intentionally
+# excluded because InnoDB silently downgrades it to NO ACTION.
+_VALID_ONDELETE_ACTIONS: frozenset[str] = frozenset(
+ {"CASCADE", "SET NULL", "RESTRICT", "NO ACTION"}
+)
+
+
+def _enforce_not_null_for_sqlite(
+ batch_op: BatchOperations, t: AssociationTable, conn: Connection
+) -> None:
+ """Force ``NOT NULL`` on the FK columns post-PK-promotion on SQLite only.
+
+ SQLite has a long-standing quirk: composite ``PRIMARY KEY`` does not
+ promote constituent columns to ``NOT NULL`` (only ``INTEGER PRIMARY KEY``
+ does). PostgreSQL and MySQL implicitly promote the PK columns to
+ ``NOT NULL`` when the constraint is added, making the explicit
+ ``alter_column`` redundant there.
+
+ Skipping the ``alter_column`` on MySQL is also functionally required:
+ MySQL 8 rejects ``ALTER COLUMN`` on a column that participates in a
+ foreign key constraint with ``ERROR 1832 (HY000): Cannot change column
+ 'X': used in a foreign key constraint 'Y'`` whenever the table has
+ data — even when the only change is ``NULL`` → ``NOT NULL`` and the
+ column is already part of a freshly-added composite primary key (which
+ InnoDB has just made implicitly ``NOT NULL`` anyway). The error fires
+ on populated tables but not on empty ones, which is why CI's
+ ``test-mysql`` shard (fresh schema) didn't catch this and a real
+ production-shaped install does.
+
+ Only SQLite still needs the explicit step, and SQLite has no FK
+ enforcement objection.
+ """
+ if conn.dialect.name == "sqlite":
+ batch_op.alter_column(t.fk1, existing_type=sa.Integer, nullable=False)
+ batch_op.alter_column(t.fk2, existing_type=sa.Integer, nullable=False)
+
+
+def upgrade() -> None:
+ conn = op.get_bind()
+ _check_no_external_fks_to_id(conn)
+ insp = inspect(conn)
+
+ for t in AFFECTED_TABLES:
+ # Resumability guard: on MySQL every DDL statement auto-commits, so
+ # a failure at table N of 8 leaves tables 1..N-1 already converted
+ # while ``alembic_version`` is still un-stamped. Without this guard
+ # a re-run would fail at table 1 (``drop_column("id")`` on a table
+ # that no longer has ``id``), and ``downgrade`` can't run either
+ # (the revision was never stamped) — recovery would need manual
+ # surgery. A converted table is identified by the absent ``id``
+ # column; skipping it makes re-running the upgrade safe on every
+ # dialect (Postgres/SQLite wrap the migration in a transaction, so
+ # the guard is simply never hit there).
+ if "id" not in {c["name"] for c in insp.get_columns(t.name)}:
+ logger.info(
+ "%s: already converted (no surrogate id column); skipping",
+ t.name,
+ )
+ continue
+
+ # Run NULL-FK cleanup unconditionally: it is a no-op DELETE on tables
+ # whose FK columns are already NOT NULL (cheap), and skipping it on a
+ # table whose FK was nullable would leave the PK-add to fail with a
+ # cryptic constraint violation. Cf. ``TABLES_WITH_NULLABLE_FKS`` above
+ # for documentation of which tables are known to have nullable FKs.
+ _delete_null_fk_rows(conn, t)
+ _dedupe_by_min_id(conn, t)
+ _assert_no_duplicates(conn, t)
+
+ # Two tables (``dashboard_slices``, ``report_schedule_user``)
+ # carry a redundant ``UNIQUE(fk1, fk2)`` that the composite PK
+ # subsumes. Three dialect-specific paths:
+ #
+ # * **PostgreSQL** — the UNIQUE constraint has a stable
+ # reflected name (Postgres default convention), so we
+ # ``DROP CONSTRAINT`` by name and then run the structural
+ # change as direct ALTER. This avoids the full-table copy
+ # that ``recreate="always"`` would trigger
+ # (``CREATE TABLE AS SELECT → DROP → RENAME``), holding
+ # ``ACCESS EXCLUSIVE`` only for the (much shorter) PK
+ # index build instead of the full copy duration.
+ #
+ # * **MySQL** — InnoDB binds the FK constraints to the
+ # redundant UNIQUE's underlying index for back-reference,
+ # so a direct ``DROP CONSTRAINT`` of the UNIQUE raises
+ # ``ERROR 1553``. Use ``recreate="always"`` to rebuild the
+ # table without the UNIQUE; drop the FKs first to dodge
+ # the ``ERROR 1826`` (duplicate FK constraint name) that
+ # the temp-table phase would otherwise provoke. The FKs
+ # are re-created automatically as part of ``copy_from``.
+ #
+ # * **SQLite** — unnamed UNIQUE constraints reflect with
+ # ``name=None`` and can't be dropped by name. Use
+ # ``recreate="always"`` + ``copy_from`` (omits UNIQUE).
+ # SQLite always rebuilds for PK changes anyway, so the
+ # recreate isn't extra cost there.
+ if t.name in TABLES_WITH_PRE_EXISTING_UNIQUE:
+ if conn.dialect.name == "postgresql":
+ _drop_redundant_unique_by_name(conn, insp, t)
+ with op.batch_alter_table(t.name) as batch_op:
+ batch_op.drop_column("id")
+ batch_op.create_primary_key(f"pk_{t.name}", [t.fk1, t.fk2])
+ _enforce_not_null_for_sqlite(batch_op, t, conn)
+ else:
+ # Capture the FK list BEFORE dropping: the copy_from table
+ # below must embed these constraints, and re-reflecting
+ # after the drop only works via the Inspector's
+ # per-instance info_cache (see _build_pre_upgrade_table).
+ pre_drop_fks = insp.get_foreign_keys(t.name)
+ if conn.dialect.name == "mysql":
+ for fk in pre_drop_fks:
+ if fk_name := fk.get("name"):
+ op.drop_constraint(fk_name, t.name, type_="foreignkey")
+ with op.batch_alter_table(
+ t.name,
+ recreate="always",
+ copy_from=_build_pre_upgrade_table(insp, t, fks=pre_drop_fks),
+ ) as batch_op:
+ batch_op.drop_column("id")
+ batch_op.create_primary_key(f"pk_{t.name}", [t.fk1, t.fk2])
+ _enforce_not_null_for_sqlite(batch_op, t, conn)
+ else:
+ with op.batch_alter_table(t.name) as batch_op:
+ batch_op.drop_column("id")
+ batch_op.create_primary_key(f"pk_{t.name}", [t.fk1, t.fk2])
+ _enforce_not_null_for_sqlite(batch_op, t, conn)
+
+
+def downgrade() -> None:
+ # Inverse order: undo upgrade transformations from last-applied to
+ # first-applied. Within each table, drop the composite PK, restore the
+ # surrogate ``id`` column, and re-add the original ``UNIQUE`` constraint
+ # on the two tables that previously carried one.
+ #
+ # Note: FK columns remain NOT NULL after downgrade (intentional asymmetry
+ # — see UPDATING.md). Restoring the original nullable state would require
+ # an explicit ``alter_column`` per FK per table for no operator value;
+ # junction-table NULL FKs were always meaningless under ``secondary=``
+ # semantics.
+ # The downgrade names the restored PK ``<table>_pkey`` (matching Postgres'
+ # default constraint-naming convention, which was the original constraint
+ # name before this migration ran) so a downgrade-then-upgrade round-trip
+ # doesn't collide on the upgrade's ``pk_<table>`` name.
+ #
+ # Adding a NOT NULL ``id`` column to a table with existing rows requires
+ # a default that fires on the existing rows. ``sa.Identity()`` (Postgres
+ # 10+ / MySQL 8+) and ``sa.Sequence`` (with explicit nextval) both
+ # backfill existing rows during ALTER TABLE; bare ``autoincrement=True``
+ # does not. ``Identity`` is the modern portable choice.
+ conn = op.get_bind()
+ insp = inspect(conn)
+ is_mysql = conn.dialect.name == "mysql"
+ for t in reversed(AFFECTED_TABLES):
+ if is_mysql:
+ _downgrade_mysql_table(insp, t)
+ else:
+ with op.batch_alter_table(t.name) as batch_op:
+ batch_op.drop_constraint(f"pk_{t.name}", type_="primary")
+ batch_op.add_column(
+ sa.Column(
+ "id",
+ sa.Integer,
+ sa.Identity(always=False),
+ nullable=False,
+ )
+ )
+ batch_op.create_primary_key(f"{t.name}_pkey", ["id"])
+ if t.name in TABLES_WITH_PRE_EXISTING_UNIQUE:
+ batch_op.create_unique_constraint(
+ f"uq_{t.name}_{t.fk1}_{t.fk2}", [t.fk1, t.fk2]
+ )
+
+
+def _downgrade_mysql_table(
+ insp: sa.engine.reflection.Inspector, t: AssociationTable
+) -> None:
+ """MySQL-specific downgrade for one table.
+
+ Two MySQL quirks force a dialect-specific path here:
+
+ 1. **ERROR 1553 — ``Cannot drop index 'PRIMARY': needed in a foreign
+ key constraint``**. InnoDB uses the composite PK index to back the
+ FK on the leftmost column. Dropping the PK before the FKs orphans
+ that backing index. PostgreSQL and SQLite create separate indexes
+ for FK columns and don't need this dance. We drop the FKs first
+ and re-add them after the structural change.
+
+ 2. **``Identity(always=False)`` on a non-PK column add does not emit
+ ``AUTO_INCREMENT`` on MySQL.** SQLAlchemy 1.4 only emits
+ ``AUTO_INCREMENT`` when the column has both ``Identity()`` and
+ ``primary_key=True`` at create time. Our portable path adds the
+ column first, then creates the PK separately — which works on
+ Postgres (the column gets ``GENERATED BY DEFAULT AS IDENTITY``)
+ and SQLite (``INTEGER PRIMARY KEY`` becomes a rowid alias) but
+ leaves MySQL without auto-generation, so existing rows can't be
+ backfilled and future ``INSERT`` statements fail with
+ ``Field 'id' doesn't have a default value``. The combined
+ ``DROP PRIMARY KEY, ADD COLUMN AUTO_INCREMENT, ADD PRIMARY KEY``
+ in a single ALTER statement is the canonical MySQL idiom: MySQL
+ backfills existing rows with sequential values and the column
+ remains auto-incrementing for future inserts.
+
+ Raw SQL is unavoidable here — there is no SQLAlchemy core equivalent
+ for the combined-ALTER form, and the constitution allows raw SQL for
+ dialect-specific DDL with no programmatic equivalent (preferring
+ triple-quoted strings for legibility).
+
+ Belt-and-braces guard: ``t.name`` is interpolated as a backtick-quoted
+ identifier in the ALTER statements below. The value comes from
+ ``AFFECTED_TABLES`` (a module-level literal), so SQL injection is
+ structurally precluded. The explicit ``allowed`` check here makes
+ that invariant load-bearing rather than implicit, so a future
+ refactor that loosens the call-site can't slip past review.
+ """
+ allowed = {a.name for a in AFFECTED_TABLES}
+ if t.name not in allowed:
+ raise RuntimeError(
+ f"Refusing to ALTER unknown table {t.name!r}: "
+ f"only AFFECTED_TABLES entries may flow through this path."
+ )
+
+ fks = insp.get_foreign_keys(t.name)
+
+ for fk in fks:
+ if fk_name := fk.get("name"):
+ op.execute(f"ALTER TABLE `{t.name}` DROP FOREIGN KEY `{fk_name}`")
Review Comment:
**Suggestion:** Replace raw SQL foreign-key drops with the shared migration FK helper utilities to keep migration behavior aligned with database-compatibility abstractions. [custom_rule]
**Severity Level:** Minor ⚠️
<details>
<summary><b>Why it matters? 🤔 </b></summary>
The shared migration utilities provide dialect-aware foreign-key dropping helpers, but this MySQL path uses raw SQL `ALTER TABLE ... DROP FOREIGN KEY` instead. Because the helper exists, the code fits the rule that prefers shared compatibility abstractions over manual raw DDL.
</details>
[](https://app.codeant.ai/fix-in-ide?tool=cursor&prompt_id=c2f5cdcc725f416dabc483daf682be1d&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset) [](https://app.codeant.ai/fix-in-ide?tool=vscode-claude&prompt_id=c2f5cdcc725f416dabc483daf682be1d&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset)
*(Use Cmd/Ctrl + Click for best experience)*
<details>
<summary><b>Prompt for AI Agent 🤖 </b></summary>
```mdx
This is a comment left during a code review.
**Path:** superset/migrations/versions/2026-05-01_23-36_2bee73611e32_composite_pk_association_tables.py
**Line:** 528:530
**Comment:**
*Custom Rule: Replace raw SQL foreign-key drops with the shared migration FK helper utilities to keep migration behavior aligned with database-compatibility abstractions.
Validate the correctness of the flagged issue. If correct, How can I resolve this? If you propose a fix, implement it and please make it concise.
Once fix is implemented, also check other comments on the same PR, and ask user if the user wants to fix the rest of the comments as well. if said yes, then fetch all the comments validate the correctness and implement a minimal fix
```
</details>
<a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F41076&comment_hash=f97404a9add27705395fa6bd52212d02bf3e071a95ca88f46ae81a057a20bb83&reaction=like'>👍</a> | <a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F41076&comment_hash=f97404a9add27705395fa6bd52212d02bf3e071a95ca88f46ae81a057a20bb83&reaction=dislike'>👎</a>
##########
superset/migrations/versions/2026-05-28_19-50_56cd24c07170_add_versioning_tables.py:
##########
@@ -0,0 +1,562 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""add_versioning_tables
+
+Creates the full schema backing entity versioning in a single
+migration:
+
+1. ``version_transaction`` — audit log keyed by Continuum's per-flush
+ transaction id (plus a Postgres-specific id sequence).
+2. **Parent shadow tables** mirroring each versioned entity's columns:
+ ``dashboards_version`` / ``slices_version`` / ``tables_version``.
+3. ``version_changes`` — field-level diff log keyed to a
+ ``(transaction, entity)`` pair; each row describes one atomic change
+ (one field or one child-collection element) that occurred during a
+ save.
+4. **Child shadow tables** for the collections Continuum auto-registers
+ when ``__versioned__`` is applied to ``TableColumn`` / ``SqlMetric``
+ and the ``slices`` exclude is removed from
+ ``Dashboard.__versioned__``: ``table_columns_version`` /
+ ``sql_metrics_version`` / ``dashboard_slices_version``.
+
+All shadow tables follow the validity-strategy shape (mirrored columns
++ ``transaction_id`` / ``end_transaction_id`` / ``operation_type``
+bookkeeping with FKs to ``version_transaction.id``). The current
+version row has ``end_transaction_id = NULL``.
+
+This migration replaces three iterative migrations from the spike phase
+(``56cd24c07170``, ``e1f3c5a7b9d0``, ``f7a2b3c4d5e6``) that captured the
+same schema in three steps as the feature was developed. Compacting
+gives downstream operators one migration to apply / reverse and one
+review surface. The ``revision`` hash is reused from the original first
+migration so anyone still tracking the chain by that hash lands on the
+same logical change set.
+
+Generated by hand because the current Continuum + Alembic-autogenerate
+interaction trips on the renamed ``transaction`` -> ``version_transaction``
+table key (``KeyError`` lookups in ``table_key_to_table``). Column
+inventories were sourced from the live model ``__table__`` definitions
+and ``version_class(...).__table__`` / Continuum association metadata.
+
+Primary key choice. Both ``version_transaction.id`` and
+``version_changes.id`` are ``BigInteger`` autoincrement — a deliberate
+carveout from the project's UUID-PK convention for new models (see
+``CLAUDE.md`` §"UUID Migration"). ``version_transaction`` is keyed
+externally by SQLAlchemy-Continuum via
+``nextval('version_transaction_id_seq')`` on every INSERT; matching
+that contract is required for ``versioning_manager`` to function.
+``version_changes`` follows the same shape because the user-facing
+identity is the ``(transaction_id, entity_kind, entity_id, sequence)``
+composite unique key, not the row id; the API surfaces a deterministic
+UUIDv5 ``version_uuid`` derived from ``entity.uuid`` and
+``transaction_id`` for stable external references.
+
+Revision ID: 56cd24c07170
+Revises: 2bee73611e32
+Create Date: 2026-05-28 19:50:00.000000
+
+"""
+
+from __future__ import annotations
+
+import sqlalchemy as sa
+from alembic import op
+from sqlalchemy_utils import UUIDType
+
+from superset.utils.core import MediumText
+
+revision = "56cd24c07170"
+# Stacked on the composite-PK association-tables change (2bee73611e32) so the
+# Continuum shadow tables this migration creates can mirror the
+# composite-PK shape of the live association tables. If that change
+# is removed from the stack, this should be reverted to "ce6bd21901ab".
+down_revision = "2bee73611e32"
+
+
+def upgrade() -> None:
+ bind = op.get_bind()
+
+ # ------------------------------------------------------------------
+ # version_transaction
+ #
+ # Audit log for each versioning event. Continuum emits
+ # ``nextval('version_transaction_id_seq')`` on every INSERT, so the
+ # sequence must exist before the table on Postgres. SQLite/MySQL
+ # ignore the explicit CREATE SEQUENCE (they auto-increment natively).
+ # ------------------------------------------------------------------
+ if bind.dialect.name == "postgresql":
+ op.execute("CREATE SEQUENCE IF NOT EXISTS version_transaction_id_seq")
+
+ op.create_table(
+ "version_transaction",
+ sa.Column(
+ "id",
+ sa.BigInteger(),
+ sa.Sequence("version_transaction_id_seq"),
+ primary_key=True,
+ autoincrement=True,
+ nullable=False,
+ ),
+ sa.Column("issued_at", sa.DateTime(), nullable=True),
+ sa.Column("remote_addr", sa.String(50), nullable=True),
+ sa.Column("user_id", sa.Integer(), nullable=True),
+ # ``action_kind`` carries the high-level avenue that produced
+ # this transaction (``restore`` / ``import`` / ``clone``).
+ # ``NULL`` is the default "ordinary save" — most rows leave
+ # this empty. Commands set
+ # ``session.info["_versioning_action_kind"]`` before commit;
+ # the change-record listener stamps the value here. Parallel
+ # to ``version_changes.entity_kind`` and ``version_changes.kind``
+ # — the schema's third ``*_kind`` column, at transaction scope.
+ sa.Column("action_kind", sa.String(32), nullable=True),
+ )
+
+ if bind.dialect.name == "postgresql":
+ op.execute(
+ "ALTER SEQUENCE version_transaction_id_seq OWNED BY version_transaction.id"
+ )
+
+ # ------------------------------------------------------------------
+ # dashboards_version
+ # ------------------------------------------------------------------
+ op.create_table(
+ "dashboards_version",
+ sa.Column("uuid", UUIDType(binary=True), nullable=True),
+ sa.Column("id", sa.Integer(), nullable=False),
+ sa.Column("dashboard_title", sa.String(500), nullable=True),
+ # ``MediumText()`` mirrors the live column type — on MySQL plain
+ # ``TEXT`` caps at 64 KB, which large dashboards exceed; an
+ # oversized live write would then fail the shadow INSERT under
+ # ``STRICT_TRANS_TABLES`` (or silently truncate without it) and
+ # corrupt the history. Postgres ``TEXT`` is unbounded and SQLite
+ # ignores the length annotation so this is MySQL-driven.
+ sa.Column("position_json", MediumText(), nullable=True),
+ sa.Column("description", sa.Text(), nullable=True),
+ sa.Column("css", MediumText(), nullable=True),
+ sa.Column("theme_id", sa.Integer(), nullable=True),
+ sa.Column("certified_by", sa.Text(), nullable=True),
+ sa.Column("certification_details", sa.Text(), nullable=True),
+ sa.Column("json_metadata", MediumText(), nullable=True),
+ sa.Column("slug", sa.String(255), nullable=True),
+ sa.Column("published", sa.Boolean(), nullable=True),
+ sa.Column("is_managed_externally", sa.Boolean(), nullable=True),
+ sa.Column("external_url", sa.Text(), nullable=True),
+ sa.Column("transaction_id", sa.BigInteger(), nullable=False),
+ sa.Column("end_transaction_id", sa.BigInteger(), nullable=True),
+ sa.Column("operation_type", sa.SmallInteger(), nullable=False),
+ sa.PrimaryKeyConstraint("id", "transaction_id"),
+ sa.ForeignKeyConstraint(
+ ["transaction_id"],
+ ["version_transaction.id"],
+ name="fk_dashboards_version_transaction_id",
+ ),
+ sa.ForeignKeyConstraint(
+ ["end_transaction_id"],
+ ["version_transaction.id"],
+ name="fk_dashboards_version_end_transaction_id",
+ ),
+ )
+ op.create_index(
Review Comment:
**Suggestion:** Use the shared index helper from `superset.migrations.shared.utils` instead of direct `op.create_index` calls for compatibility-aware migrations. [custom_rule]
**Severity Level:** Minor ⚠️
<details>
<summary><b>Why it matters? 🤔 </b></summary>
superset.migrations.shared.utils provides a create_index helper, so the migration is using raw Alembic index creation instead of the compatibility-aware utility required by the rule.
</details>
[](https://app.codeant.ai/fix-in-ide?tool=cursor&prompt_id=c1985729a2d24e81826ce4ec132b9829&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset) [](https://app.codeant.ai/fix-in-ide?tool=vscode-claude&prompt_id=c1985729a2d24e81826ce4ec132b9829&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset)
*(Use Cmd/Ctrl + Click for best experience)*
<details>
<summary><b>Prompt for AI Agent 🤖 </b></summary>
```mdx
This is a comment left during a code review.
**Path:** superset/migrations/versions/2026-05-28_19-50_56cd24c07170_add_versioning_tables.py
**Line:** 173:173
**Comment:**
*Custom Rule: Use the shared index helper from `superset.migrations.shared.utils` instead of direct `op.create_index` calls for compatibility-aware migrations.
Validate the correctness of the flagged issue. If correct, How can I resolve this? If you propose a fix, implement it and please make it concise.
Once fix is implemented, also check other comments on the same PR, and ask user if the user wants to fix the rest of the comments as well. if said yes, then fetch all the comments validate the correctness and implement a minimal fix
```
</details>
<a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F41076&comment_hash=9f1f98896f68324030809d1c6fb03189c5db5f3d83a37af06bb01f3eb121ac25&reaction=like'>👍</a> | <a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F41076&comment_hash=9f1f98896f68324030809d1c6fb03189c5db5f3d83a37af06bb01f3eb121ac25&reaction=dislike'>👎</a>
##########
superset/migrations/versions/2026-05-28_19-50_56cd24c07170_add_versioning_tables.py:
##########
@@ -0,0 +1,562 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""add_versioning_tables
+
+Creates the full schema backing entity versioning in a single
+migration:
+
+1. ``version_transaction`` — audit log keyed by Continuum's per-flush
+ transaction id (plus a Postgres-specific id sequence).
+2. **Parent shadow tables** mirroring each versioned entity's columns:
+ ``dashboards_version`` / ``slices_version`` / ``tables_version``.
+3. ``version_changes`` — field-level diff log keyed to a
+ ``(transaction, entity)`` pair; each row describes one atomic change
+ (one field or one child-collection element) that occurred during a
+ save.
+4. **Child shadow tables** for the collections Continuum auto-registers
+ when ``__versioned__`` is applied to ``TableColumn`` / ``SqlMetric``
+ and the ``slices`` exclude is removed from
+ ``Dashboard.__versioned__``: ``table_columns_version`` /
+ ``sql_metrics_version`` / ``dashboard_slices_version``.
+
+All shadow tables follow the validity-strategy shape (mirrored columns
++ ``transaction_id`` / ``end_transaction_id`` / ``operation_type``
+bookkeeping with FKs to ``version_transaction.id``). The current
+version row has ``end_transaction_id = NULL``.
+
+This migration replaces three iterative migrations from the spike phase
+(``56cd24c07170``, ``e1f3c5a7b9d0``, ``f7a2b3c4d5e6``) that captured the
+same schema in three steps as the feature was developed. Compacting
+gives downstream operators one migration to apply / reverse and one
+review surface. The ``revision`` hash is reused from the original first
+migration so anyone still tracking the chain by that hash lands on the
+same logical change set.
+
+Generated by hand because the current Continuum + Alembic-autogenerate
+interaction trips on the renamed ``transaction`` -> ``version_transaction``
+table key (``KeyError`` lookups in ``table_key_to_table``). Column
+inventories were sourced from the live model ``__table__`` definitions
+and ``version_class(...).__table__`` / Continuum association metadata.
+
+Primary key choice. Both ``version_transaction.id`` and
+``version_changes.id`` are ``BigInteger`` autoincrement — a deliberate
+carveout from the project's UUID-PK convention for new models (see
+``CLAUDE.md`` §"UUID Migration"). ``version_transaction`` is keyed
+externally by SQLAlchemy-Continuum via
+``nextval('version_transaction_id_seq')`` on every INSERT; matching
+that contract is required for ``versioning_manager`` to function.
+``version_changes`` follows the same shape because the user-facing
+identity is the ``(transaction_id, entity_kind, entity_id, sequence)``
+composite unique key, not the row id; the API surfaces a deterministic
+UUIDv5 ``version_uuid`` derived from ``entity.uuid`` and
+``transaction_id`` for stable external references.
+
+Revision ID: 56cd24c07170
+Revises: 2bee73611e32
+Create Date: 2026-05-28 19:50:00.000000
+
+"""
+
+from __future__ import annotations
+
+import sqlalchemy as sa
+from alembic import op
+from sqlalchemy_utils import UUIDType
+
+from superset.utils.core import MediumText
+
+revision = "56cd24c07170"
+# Stacked on the composite-PK association-tables change (2bee73611e32) so the
+# Continuum shadow tables this migration creates can mirror the
+# composite-PK shape of the live association tables. If that change
+# is removed from the stack, this should be reverted to "ce6bd21901ab".
+down_revision = "2bee73611e32"
+
+
+def upgrade() -> None:
+ bind = op.get_bind()
+
+ # ------------------------------------------------------------------
+ # version_transaction
+ #
+ # Audit log for each versioning event. Continuum emits
+ # ``nextval('version_transaction_id_seq')`` on every INSERT, so the
+ # sequence must exist before the table on Postgres. SQLite/MySQL
+ # ignore the explicit CREATE SEQUENCE (they auto-increment natively).
+ # ------------------------------------------------------------------
+ if bind.dialect.name == "postgresql":
+ op.execute("CREATE SEQUENCE IF NOT EXISTS version_transaction_id_seq")
+
+ op.create_table(
+ "version_transaction",
+ sa.Column(
+ "id",
+ sa.BigInteger(),
+ sa.Sequence("version_transaction_id_seq"),
+ primary_key=True,
+ autoincrement=True,
+ nullable=False,
+ ),
+ sa.Column("issued_at", sa.DateTime(), nullable=True),
+ sa.Column("remote_addr", sa.String(50), nullable=True),
+ sa.Column("user_id", sa.Integer(), nullable=True),
+ # ``action_kind`` carries the high-level avenue that produced
+ # this transaction (``restore`` / ``import`` / ``clone``).
+ # ``NULL`` is the default "ordinary save" — most rows leave
+ # this empty. Commands set
+ # ``session.info["_versioning_action_kind"]`` before commit;
+ # the change-record listener stamps the value here. Parallel
+ # to ``version_changes.entity_kind`` and ``version_changes.kind``
+ # — the schema's third ``*_kind`` column, at transaction scope.
+ sa.Column("action_kind", sa.String(32), nullable=True),
+ )
+
+ if bind.dialect.name == "postgresql":
+ op.execute(
+ "ALTER SEQUENCE version_transaction_id_seq OWNED BY version_transaction.id"
+ )
+
+ # ------------------------------------------------------------------
+ # dashboards_version
+ # ------------------------------------------------------------------
+ op.create_table(
+ "dashboards_version",
+ sa.Column("uuid", UUIDType(binary=True), nullable=True),
+ sa.Column("id", sa.Integer(), nullable=False),
+ sa.Column("dashboard_title", sa.String(500), nullable=True),
+ # ``MediumText()`` mirrors the live column type — on MySQL plain
+ # ``TEXT`` caps at 64 KB, which large dashboards exceed; an
+ # oversized live write would then fail the shadow INSERT under
+ # ``STRICT_TRANS_TABLES`` (or silently truncate without it) and
+ # corrupt the history. Postgres ``TEXT`` is unbounded and SQLite
+ # ignores the length annotation so this is MySQL-driven.
+ sa.Column("position_json", MediumText(), nullable=True),
+ sa.Column("description", sa.Text(), nullable=True),
+ sa.Column("css", MediumText(), nullable=True),
+ sa.Column("theme_id", sa.Integer(), nullable=True),
+ sa.Column("certified_by", sa.Text(), nullable=True),
+ sa.Column("certification_details", sa.Text(), nullable=True),
+ sa.Column("json_metadata", MediumText(), nullable=True),
+ sa.Column("slug", sa.String(255), nullable=True),
+ sa.Column("published", sa.Boolean(), nullable=True),
+ sa.Column("is_managed_externally", sa.Boolean(), nullable=True),
+ sa.Column("external_url", sa.Text(), nullable=True),
+ sa.Column("transaction_id", sa.BigInteger(), nullable=False),
+ sa.Column("end_transaction_id", sa.BigInteger(), nullable=True),
+ sa.Column("operation_type", sa.SmallInteger(), nullable=False),
+ sa.PrimaryKeyConstraint("id", "transaction_id"),
+ sa.ForeignKeyConstraint(
+ ["transaction_id"],
+ ["version_transaction.id"],
+ name="fk_dashboards_version_transaction_id",
+ ),
+ sa.ForeignKeyConstraint(
+ ["end_transaction_id"],
+ ["version_transaction.id"],
+ name="fk_dashboards_version_end_transaction_id",
+ ),
+ )
+ op.create_index(
+ "ix_dashboards_version_end_transaction_id",
+ "dashboards_version",
+ ["end_transaction_id"],
+ )
+ op.create_index(
+ "ix_dashboards_version_operation_type",
+ "dashboards_version",
+ ["operation_type"],
+ )
+ op.create_index(
+ "ix_dashboards_version_transaction_id",
+ "dashboards_version",
+ ["transaction_id"],
+ )
+
+ # ------------------------------------------------------------------
+ # slices_version (Charts)
+ # ------------------------------------------------------------------
+ op.create_table(
+ "slices_version",
+ sa.Column("uuid", UUIDType(binary=True), nullable=True),
+ sa.Column("id", sa.Integer(), nullable=False),
+ sa.Column("slice_name", sa.String(250), nullable=True),
+ sa.Column("datasource_id", sa.Integer(), nullable=True),
+ sa.Column("datasource_type", sa.String(200), nullable=True),
+ sa.Column("datasource_name", sa.String(2000), nullable=True),
+ sa.Column("viz_type", sa.String(250), nullable=True),
+ sa.Column("params", MediumText(), nullable=True),
+ sa.Column("description", sa.Text(), nullable=True),
+ sa.Column("cache_timeout", sa.Integer(), nullable=True),
+ sa.Column("certified_by", sa.Text(), nullable=True),
+ sa.Column("certification_details", sa.Text(), nullable=True),
+ sa.Column("is_managed_externally", sa.Boolean(), nullable=True),
+ sa.Column("external_url", sa.Text(), nullable=True),
+ sa.Column("transaction_id", sa.BigInteger(), nullable=False),
+ sa.Column("end_transaction_id", sa.BigInteger(), nullable=True),
+ sa.Column("operation_type", sa.SmallInteger(), nullable=False),
+ sa.PrimaryKeyConstraint("id", "transaction_id"),
+ sa.ForeignKeyConstraint(
+ ["transaction_id"],
+ ["version_transaction.id"],
+ name="fk_slices_version_transaction_id",
+ ),
+ sa.ForeignKeyConstraint(
+ ["end_transaction_id"],
+ ["version_transaction.id"],
+ name="fk_slices_version_end_transaction_id",
+ ),
+ )
+ op.create_index(
+ "ix_slices_version_end_transaction_id",
+ "slices_version",
+ ["end_transaction_id"],
+ )
+ op.create_index(
+ "ix_slices_version_operation_type",
+ "slices_version",
+ ["operation_type"],
+ )
+ op.create_index(
+ "ix_slices_version_transaction_id",
+ "slices_version",
+ ["transaction_id"],
+ )
+
+ # ------------------------------------------------------------------
+ # tables_version (SqlaTable / Datasets)
+ # ------------------------------------------------------------------
+ op.create_table(
+ "tables_version",
+ sa.Column("uuid", UUIDType(binary=True), nullable=True),
+ sa.Column("id", sa.Integer(), nullable=False),
+ sa.Column("description", sa.Text(), nullable=True),
+ sa.Column("default_endpoint", sa.Text(), nullable=True),
+ sa.Column("is_featured", sa.Boolean(), nullable=True),
+ sa.Column("filter_select_enabled", sa.Boolean(), nullable=True),
+ sa.Column("offset", sa.Integer(), nullable=True),
+ sa.Column("cache_timeout", sa.Integer(), nullable=True),
+ sa.Column("params", sa.String(1000), nullable=True),
+ sa.Column("is_managed_externally", sa.Boolean(), nullable=True),
+ sa.Column("external_url", sa.Text(), nullable=True),
+ sa.Column("table_name", sa.String(250), nullable=True),
+ sa.Column("main_dttm_col", sa.String(250), nullable=True),
+ sa.Column("currency_code_column", sa.String(250), nullable=True),
+ sa.Column("database_id", sa.Integer(), nullable=True),
+ sa.Column("fetch_values_predicate", sa.Text(), nullable=True),
+ sa.Column("schema", sa.String(255), nullable=True),
+ sa.Column("catalog", sa.String(256), nullable=True),
+ sa.Column("sql", MediumText(), nullable=True),
+ sa.Column("is_sqllab_view", sa.Boolean(), nullable=True),
+ sa.Column("template_params", sa.Text(), nullable=True),
+ sa.Column("extra", sa.Text(), nullable=True),
+ sa.Column("normalize_columns", sa.Boolean(), nullable=True),
+ sa.Column("always_filter_main_dttm", sa.Boolean(), nullable=True),
+ sa.Column("folders", sa.JSON(), nullable=True),
+ sa.Column("transaction_id", sa.BigInteger(), nullable=False),
+ sa.Column("end_transaction_id", sa.BigInteger(), nullable=True),
+ sa.Column("operation_type", sa.SmallInteger(), nullable=False),
+ sa.PrimaryKeyConstraint("id", "transaction_id"),
+ sa.ForeignKeyConstraint(
+ ["transaction_id"],
+ ["version_transaction.id"],
+ name="fk_tables_version_transaction_id",
+ ),
+ sa.ForeignKeyConstraint(
+ ["end_transaction_id"],
+ ["version_transaction.id"],
+ name="fk_tables_version_end_transaction_id",
+ ),
+ )
+ op.create_index(
+ "ix_tables_version_end_transaction_id",
+ "tables_version",
+ ["end_transaction_id"],
+ )
+ op.create_index(
+ "ix_tables_version_operation_type",
+ "tables_version",
+ ["operation_type"],
+ )
+ op.create_index(
+ "ix_tables_version_transaction_id",
+ "tables_version",
+ ["transaction_id"],
+ )
+
+ # ------------------------------------------------------------------
+ # version_changes
+ #
+ # Field-level diff log keyed to a (transaction, entity) pair. Each
+ # row describes one atomic change (one field or one child-collection
+ # element) that occurred to one entity during a save.
+ #
+ # ``(entity_kind, entity_id)`` is a polymorphic reference: depending
+ # on ``entity_kind`` (``"chart"`` / ``"dashboard"`` / ``"dataset"``)
+ # the ``entity_id`` is the integer PK on ``slices`` / ``dashboards`` /
+ # ``tables`` respectively. SQL has no native polymorphic FK, so the
+ # constraint is intentionally omitted — cleanup relies on the
+ # ``CASCADE`` from ``version_transaction.id`` plus command-layer
+ # ordering for entity deletes (the command that hard-deletes the
+ # entity runs inside the same transaction that prunes its history).
+ # A bare ``DELETE FROM <entity_table> WHERE id = X`` outside that
+ # transactional boundary leaves orphan ``version_changes`` rows
+ # whose ``entity_id`` references a vanished row — the read-side
+ # tombstone-state lookup handles this gracefully.
+ # ------------------------------------------------------------------
+ op.create_table(
+ "version_changes",
+ sa.Column(
+ "id",
+ sa.BigInteger(),
+ primary_key=True,
+ autoincrement=True,
+ nullable=False,
+ ),
+ sa.Column(
+ "transaction_id",
+ sa.BigInteger(),
+ sa.ForeignKey("version_transaction.id", ondelete="CASCADE"),
+ nullable=False,
+ ),
+ sa.Column(
+ "entity_kind",
+ sa.String(length=32),
+ nullable=False,
+ ),
+ sa.Column(
+ "entity_id",
+ sa.Integer(),
+ nullable=False,
+ ),
+ sa.Column(
+ # Integer, not SmallInteger: per-entity sequence within one
+ # transaction is assigned by unbounded enumerate(); a
+ # pathological diff (e.g. a giant position_json rewrite) could
+ # overflow SmallInteger's 32767 on Postgres/MySQL.
+ "sequence",
+ sa.Integer(),
+ nullable=False,
+ ),
+ sa.Column(
+ "kind",
+ sa.String(length=32),
+ nullable=False,
+ ),
+ # ``operation`` is the per-record verb: ``add`` / ``remove`` /
+ # ``move`` / ``edit``. ``move`` only fires for layout records;
+ # the other three apply across every emit site. Made explicit
+ # so consumers don't have to infer the verb from ``from_value``
+ # / ``to_value`` null-tests or from ``path[0]`` for layout records.
+ sa.Column(
+ "operation",
+ sa.String(length=16),
+ nullable=False,
+ ),
+ sa.Column("path", sa.JSON(), nullable=False),
+ sa.Column("from_value", sa.JSON(), nullable=True),
+ sa.Column("to_value", sa.JSON(), nullable=True),
+ sa.UniqueConstraint(
+ "transaction_id",
+ "entity_kind",
+ "entity_id",
+ "sequence",
+ name="uq_version_changes_tx_entity_sequence",
+ ),
+ )
+ op.create_index(
+ "ix_version_changes_kind",
+ "version_changes",
+ ["kind"],
+ )
+ op.create_index(
+ "ix_version_changes_entity",
+ "version_changes",
+ ["entity_kind", "entity_id"],
+ )
+
+ # ------------------------------------------------------------------
+ # table_columns_version
+ # ------------------------------------------------------------------
+ op.create_table(
+ "table_columns_version",
+ sa.Column("uuid", UUIDType(binary=True), nullable=True),
+ sa.Column("id", sa.Integer(), nullable=False),
+ sa.Column("column_name", sa.String(255), nullable=True),
+ sa.Column("verbose_name", sa.String(1024), nullable=True),
+ sa.Column("is_active", sa.Boolean(), nullable=True),
+ sa.Column("type", sa.Text(), nullable=True),
+ sa.Column("advanced_data_type", sa.String(255), nullable=True),
+ sa.Column("groupby", sa.Boolean(), nullable=True),
+ sa.Column("filterable", sa.Boolean(), nullable=True),
+ sa.Column("description", MediumText(), nullable=True),
+ sa.Column("table_id", sa.Integer(), nullable=True),
+ sa.Column("is_dttm", sa.Boolean(), nullable=True),
+ sa.Column("expression", MediumText(), nullable=True),
+ sa.Column("python_date_format", sa.String(255), nullable=True),
+ sa.Column("datetime_format", sa.String(100), nullable=True),
+ sa.Column("extra", sa.Text(), nullable=True),
+ sa.Column("transaction_id", sa.BigInteger(), nullable=False),
+ sa.Column("end_transaction_id", sa.BigInteger(), nullable=True),
+ sa.Column("operation_type", sa.SmallInteger(), nullable=False),
+ sa.PrimaryKeyConstraint("id", "transaction_id"),
+ sa.ForeignKeyConstraint(
+ ["transaction_id"],
+ ["version_transaction.id"],
+ name="fk_table_columns_version_transaction_id",
+ ),
+ sa.ForeignKeyConstraint(
+ ["end_transaction_id"],
+ ["version_transaction.id"],
+ name="fk_table_columns_version_end_transaction_id",
+ ),
+ )
+ op.create_index(
+ "ix_table_columns_version_end_transaction_id",
+ "table_columns_version",
+ ["end_transaction_id"],
+ )
+ op.create_index(
+ "ix_table_columns_version_operation_type",
+ "table_columns_version",
+ ["operation_type"],
+ )
+ op.create_index(
+ "ix_table_columns_version_transaction_id",
+ "table_columns_version",
+ ["transaction_id"],
+ )
+
+ # ------------------------------------------------------------------
+ # sql_metrics_version
+ # ------------------------------------------------------------------
+ op.create_table(
+ "sql_metrics_version",
+ sa.Column("uuid", UUIDType(binary=True), nullable=True),
+ sa.Column("id", sa.Integer(), nullable=False),
+ sa.Column("metric_name", sa.String(255), nullable=True),
+ sa.Column("verbose_name", sa.String(1024), nullable=True),
+ sa.Column("metric_type", sa.String(32), nullable=True),
+ sa.Column("description", MediumText(), nullable=True),
+ sa.Column("d3format", sa.String(128), nullable=True),
+ sa.Column("currency", sa.JSON(), nullable=True),
+ sa.Column("warning_text", sa.Text(), nullable=True),
+ sa.Column("table_id", sa.Integer(), nullable=True),
+ sa.Column("expression", MediumText(), nullable=True),
+ sa.Column("extra", sa.Text(), nullable=True),
+ sa.Column("transaction_id", sa.BigInteger(), nullable=False),
+ sa.Column("end_transaction_id", sa.BigInteger(), nullable=True),
+ sa.Column("operation_type", sa.SmallInteger(), nullable=False),
+ sa.PrimaryKeyConstraint("id", "transaction_id"),
+ sa.ForeignKeyConstraint(
+ ["transaction_id"],
+ ["version_transaction.id"],
+ name="fk_sql_metrics_version_transaction_id",
+ ),
+ sa.ForeignKeyConstraint(
+ ["end_transaction_id"],
+ ["version_transaction.id"],
+ name="fk_sql_metrics_version_end_transaction_id",
+ ),
+ )
+ op.create_index(
+ "ix_sql_metrics_version_end_transaction_id",
+ "sql_metrics_version",
+ ["end_transaction_id"],
+ )
+ op.create_index(
+ "ix_sql_metrics_version_operation_type",
+ "sql_metrics_version",
+ ["operation_type"],
+ )
+ op.create_index(
+ "ix_sql_metrics_version_transaction_id",
+ "sql_metrics_version",
+ ["transaction_id"],
+ )
+
+ # ------------------------------------------------------------------
+ # dashboard_slices_version (M2M association)
+ #
+ # The live ``dashboard_slices`` table is reshaped to a
+ # composite PK on ``(dashboard_id, slice_id)`` — no surrogate ``id``.
+ # Continuum auto-mirrors the live columns into the shadow Table at
+ # ``make_versioned()`` time, so the shadow's SQLAlchemy metadata
+ # also has no ``id``. The DB shadow PK is the natural composite key
+ # plus Continuum's bookkeeping (``transaction_id``, ``operation_type``);
+ # ``operation_type`` is included because a single transaction can in
+ # principle produce both INSERT and DELETE shadows for the same
+ # ``(dashboard_id, slice_id)`` pair (slice removed and re-added in
+ # one save).
+ #
+ # If that reshape is removed from the stack, the live table reverts to
+ # carrying its surrogate ``id`` and this migration would need to
+ # match.
+ # ------------------------------------------------------------------
+ op.create_table(
+ "dashboard_slices_version",
+ sa.Column("dashboard_id", sa.Integer(), nullable=False),
+ sa.Column("slice_id", sa.Integer(), nullable=False),
+ sa.Column("transaction_id", sa.BigInteger(), nullable=False),
+ sa.Column("end_transaction_id", sa.BigInteger(), nullable=True),
+ sa.Column("operation_type", sa.SmallInteger(), nullable=False),
+ sa.PrimaryKeyConstraint(
+ "dashboard_id", "slice_id", "transaction_id", "operation_type"
+ ),
+ sa.ForeignKeyConstraint(
+ ["transaction_id"],
+ ["version_transaction.id"],
+ name="fk_dashboard_slices_version_transaction_id",
+ ),
+ sa.ForeignKeyConstraint(
+ ["end_transaction_id"],
+ ["version_transaction.id"],
+ name="fk_dashboard_slices_version_end_transaction_id",
+ ),
+ )
+ op.create_index(
+ "ix_dashboard_slices_version_end_transaction_id",
+ "dashboard_slices_version",
+ ["end_transaction_id"],
+ )
+ op.create_index(
+ "ix_dashboard_slices_version_operation_type",
+ "dashboard_slices_version",
+ ["operation_type"],
+ )
+ op.create_index(
+ "ix_dashboard_slices_version_transaction_id",
+ "dashboard_slices_version",
+ ["transaction_id"],
+ )
+
+
+def downgrade() -> None:
+ # Drop in reverse dependency order: children with FKs to
+ # ``version_transaction`` drop first; ``version_transaction`` and its
+ # sequence drop last.
+ op.drop_table("dashboard_slices_version")
Review Comment:
**Suggestion:** Use the shared table-drop helper from `superset.migrations.shared.utils` instead of direct `op.drop_table` operations. [custom_rule]
**Severity Level:** Minor ⚠️
<details>
<summary><b>Why it matters? 🤔 </b></summary>
The shared utilities include a drop_table helper, and this downgrade uses direct op.drop_table calls instead of the database-safe wrapper, matching the migration rule violation.
</details>
[](https://app.codeant.ai/fix-in-ide?tool=cursor&prompt_id=bc16b980bf1e4020a2680aae7cca9289&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset) [](https://app.codeant.ai/fix-in-ide?tool=vscode-claude&prompt_id=bc16b980bf1e4020a2680aae7cca9289&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset)
*(Use Cmd/Ctrl + Click for best experience)*
<details>
<summary><b>Prompt for AI Agent 🤖 </b></summary>
```mdx
This is a comment left during a code review.
**Path:** superset/migrations/versions/2026-05-28_19-50_56cd24c07170_add_versioning_tables.py
**Line:** 551:551
**Comment:**
*Custom Rule: Use the shared table-drop helper from `superset.migrations.shared.utils` instead of direct `op.drop_table` operations.
Validate the correctness of the flagged issue. If correct, How can I resolve this? If you propose a fix, implement it and please make it concise.
Once fix is implemented, also check other comments on the same PR, and ask user if the user wants to fix the rest of the comments as well. if said yes, then fetch all the comments validate the correctness and implement a minimal fix
```
</details>
<a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F41076&comment_hash=876063d4a6275bc31ee40cf6d74fce0cf9006151c598d1e15e188cf639109a55&reaction=like'>👍</a> | <a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F41076&comment_hash=876063d4a6275bc31ee40cf6d74fce0cf9006151c598d1e15e188cf639109a55&reaction=dislike'>👎</a>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(versioning): cross-entity version activity view [superset]
Posted by "mikebridge (via GitHub)" <gi...@apache.org>.
mikebridge commented on code in PR #41076:
URL: https://github.com/apache/superset/pull/41076#discussion_r3574959386
##########
scripts/seed_junction_load.py:
##########
@@ -0,0 +1,679 @@
+#!/usr/bin/env python3
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+# ----------------------------------------------------------------------
+# Stress-test data generator for the composite-PK migration (sc-105349).
+#
+# Bulk-inserts synthetic parent rows and many-to-many junction rows for
+# the eight association tables that the composite-PK migration touches.
+# Useful for measuring migration runtime at varying scales — run this at
+# 100K / 1M / 5M / 10M rows and time the migration at each scale to
+# verify the O(N log N) extrapolation.
+#
+# Idempotent: rerunning with the same target is a no-op; rerunning with
+# a higher target adds rows up to the new total. Batched bulk INSERTs
+# (10K rows per statement) make it fast on Postgres, MySQL, and SQLite.
+#
+# Usage (inside the Superset container):
+#
+# docker exec superset-superset-1 \\
+# /app/.venv/bin/python /app/scripts/seed_junction_load.py \\
+# --dashboard-slices 1000000 \\
+# --slice-user 100000 \\
+# --dashboard-user 100000
+#
+# Run with no flags for the defaults shown below. Use ``--dry-run`` to
+# print the planned inserts without writing anything.
+#
+# The script connects via Superset's standard ``DATABASE_*`` env vars
+# (or ``SUPERSET__SQLALCHEMY_DATABASE_URI`` if set), so it works
+# automatically inside the Superset container regardless of which
+# metadata DB backend is in use.
+
+from __future__ import annotations
+
+import argparse
+import logging
+import os
+import sys
+import time
+from contextlib import contextmanager
+from typing import Iterator
+from uuid import uuid4
+
+import sqlalchemy as sa
+from sqlalchemy.engine import Connection, Engine
+
+logger = logging.getLogger("seed_junction_load")
+
+# Bulk INSERT batch size. Larger values = fewer statements but more memory.
+BATCH = 10_000
+
+# Default per-junction-table target row counts. Tuned to mimic the shape
+# of a large multi-team Superset install. Override via CLI flags.
+DEFAULTS: dict[str, int] = {
+ "dashboard_slices": 1_000_000,
+ "slice_user": 100_000,
+ "dashboard_user": 100_000,
+ "dashboard_roles": 10_000,
+}
+
+# (junction_table, fk1_col, fk2_col, parent1_table, parent2_table)
+# parents reference id columns; we generate (fk1, fk2) pairs by sampling
+# from the parents' existing IDs.
+JUNCTIONS: list[tuple[str, str, str, str, str]] = [
+ ("dashboard_slices", "dashboard_id", "slice_id", "dashboards", "slices"),
+ ("slice_user", "user_id", "slice_id", "ab_user", "slices"),
+ ("dashboard_user", "user_id", "dashboard_id", "ab_user", "dashboards"),
+ ("dashboard_roles", "dashboard_id", "role_id", "dashboards", "ab_role"),
+]
+
+# Junction tables that originally carried ``UNIQUE(fk1, fk2)`` and therefore
+# cannot accept duplicate ``(fk1, fk2)`` pairs even on the pre-migration
+# (downgrade) schema. The other JUNCTIONS allow duplicates pre-migration.
+# Only ``dashboard_slices`` is listed: the migration's other UNIQUE table
+# (``report_schedule_user``) is not in JUNCTIONS — this script doesn't seed
+# it — so listing it here would imply coverage that doesn't exist. Add it
+# alongside a JUNCTIONS entry if that table ever gets seeded.
+JUNCTIONS_WITH_UNIQUE: set[str] = {"dashboard_slices"}
+
+
+# ----------------------------------------------------------------------
+# Connection setup
+# ----------------------------------------------------------------------
+
+
+def build_engine() -> Engine:
+ """Build a SQLAlchemy engine from Superset env vars."""
+ if uri := os.environ.get("SUPERSET__SQLALCHEMY_DATABASE_URI"):
+ logger.info("Using SUPERSET__SQLALCHEMY_DATABASE_URI from env")
+ return sa.create_engine(uri)
+
+ try:
+ dialect = os.environ["DATABASE_DIALECT"]
+ user = os.environ["DATABASE_USER"]
+ password = os.environ["DATABASE_PASSWORD"]
+ host = os.environ["DATABASE_HOST"]
+ port = os.environ["DATABASE_PORT"]
+ db = os.environ["DATABASE_DB"]
+ except KeyError as exc:
+ sys.exit(
+ f"Missing env var {exc}; either set DATABASE_DIALECT/USER/PASSWORD/"
+ f"HOST/PORT/DB or SUPERSET__SQLALCHEMY_DATABASE_URI before running."
+ )
+
+ uri = f"{dialect}://{user}:{password}@{host}:{port}/{db}"
+ logger.info(
+ "Built URI from DATABASE_* env vars (dialect=%s, host=%s)", dialect, host
+ )
+ return sa.create_engine(uri)
+
+
+# ----------------------------------------------------------------------
+# Helpers
+# ----------------------------------------------------------------------
+
+
+def uuid_value(dialect_name: str) -> bytes | str:
+ """Return a UUID in the form the active dialect expects.
+
+ MySQL stores UUIDs as ``BINARY(16)`` (16 raw bytes); Postgres has a
+ native ``UUID`` type that accepts strings; SQLite stores them as
+ BLOB/TEXT and accepts either. Branching here keeps the seed script
+ backend-agnostic without depending on Superset's custom column types.
+ """
+ if dialect_name.startswith("mysql"):
+ return uuid4().bytes
+ return str(uuid4())
+
+
+@contextmanager
+def time_phase(name: str) -> Iterator[None]:
+ """Log elapsed wall time for a named phase."""
+ start = time.monotonic()
+ logger.info("[%s] starting", name)
+ try:
+ yield
+ finally:
+ elapsed = time.monotonic() - start
+ logger.info("[%s] done in %.2fs", name, elapsed)
+
+
+def count_rows(conn: Connection, table: str) -> int:
+ return conn.scalar(sa.text(f"SELECT COUNT(*) FROM {table}")) or 0 # noqa: S608
+
+
+def existing_ids(conn: Connection, table: str, limit: int | None = None) -> list[int]:
+ sql = f"SELECT id FROM {table} ORDER BY id" # noqa: S608
+ if limit is not None:
+ sql += f" LIMIT {limit}"
+ return [row[0] for row in conn.execute(sa.text(sql))]
+
+
+# ----------------------------------------------------------------------
+# Parent seeders
+#
+# Each function ensures the named parent table has at least ``target``
+# rows by inserting synthetic ones with minimal-but-valid columns.
+# Returns nothing; subsequent code reads back IDs via ``existing_ids``.
+# ----------------------------------------------------------------------
+
+
+def seed_dashboards(conn: Connection, target: int, dry_run: bool) -> None:
+ current = count_rows(conn, "dashboards")
+ if current >= target:
+ logger.info(
+ "dashboards: %d rows (target %d) — no insert needed", current, target
+ )
+ return
+ needed = target - current
+ logger.info("dashboards: %d → %d (+%d)", current, target, needed)
+ if dry_run:
+ return
+
+ dialect = conn.engine.dialect.name
+ sql = sa.text(
+ "INSERT INTO dashboards (uuid, dashboard_title, slug, published) "
+ "VALUES (:uuid, :title, :slug, :published)"
+ )
+ for batch_start in range(0, needed, BATCH):
+ rows = [
+ {
+ "uuid": uuid_value(dialect),
+ "title": f"seed_dashboard_{current + i}",
+ "slug": f"seed-dashboard-{current + i}-{uuid4().hex[:8]}",
+ "published": False,
+ }
+ for i in range(batch_start, min(batch_start + BATCH, needed))
+ ]
+ conn.execute(sql, rows)
+ logger.info(" dashboards: inserted %d / %d", batch_start + len(rows), needed)
+
+
+def seed_dbs(conn: Connection, dry_run: bool) -> int:
+ """Ensure at least one row exists in ``dbs`` (parent of ``tables``).
+ Returns the id to use as ``database_id`` when seeding ``tables``."""
+ ids = existing_ids(conn, "dbs", limit=1)
+ if ids:
+ return ids[0]
+ if dry_run:
+ return -1 # placeholder
+ dialect = conn.engine.dialect.name
+ logger.info("dbs: inserting one synthetic database (no rows present)")
+ conn.execute(
+ sa.text(
+ "INSERT INTO dbs (uuid, database_name, sqlalchemy_uri, expose_in_sqllab) "
+ "VALUES (:uuid, :name, :uri, :expose)"
+ ),
+ {
+ "uuid": uuid_value(dialect),
+ "name": f"seed_db_{uuid4().hex[:8]}",
+ "uri": "sqlite:///seed.db",
+ "expose": False,
+ },
+ )
+ return existing_ids(conn, "dbs", limit=1)[0]
+
+
+def seed_tables(conn: Connection, target: int, dry_run: bool) -> None:
+ current = count_rows(conn, "tables")
+ if current >= target:
+ logger.info("tables: %d rows (target %d) — no insert needed", current, target)
+ return
+ needed = target - current
+ logger.info("tables: %d → %d (+%d)", current, target, needed)
+ if dry_run:
+ return
+
+ database_id = seed_dbs(conn, dry_run=False)
+ dialect = conn.engine.dialect.name
+ sql = sa.text(
+ "INSERT INTO tables (uuid, table_name, database_id) "
+ "VALUES (:uuid, :name, :db_id)"
+ )
+ for batch_start in range(0, needed, BATCH):
+ rows = [
+ {
+ "uuid": uuid_value(dialect),
+ "name": f"seed_table_{current + i}",
+ "db_id": database_id,
+ }
+ for i in range(batch_start, min(batch_start + BATCH, needed))
+ ]
+ conn.execute(sql, rows)
+ logger.info(" tables: inserted %d / %d", batch_start + len(rows), needed)
+
+
+def seed_slices(conn: Connection, target: int, dry_run: bool) -> None:
+ current = count_rows(conn, "slices")
+ if current >= target:
+ logger.info("slices: %d rows (target %d) — no insert needed", current, target)
+ return
+ needed = target - current
+ logger.info("slices: %d → %d (+%d)", current, target, needed)
+ if dry_run:
+ return
+
+ # Slices reference tables.id; ensure at least one ``tables`` row exists
+ # so the FK is satisfiable (datasource_id is nullable but we set it for
+ # realism). The migration test doesn't care, but a real Superset that
+ # re-renders these slices does.
+ seed_tables(conn, target=1, dry_run=False)
+ table_id = existing_ids(conn, "tables", limit=1)[0]
+ dialect = conn.engine.dialect.name
+ sql = sa.text(
+ "INSERT INTO slices "
+ "(uuid, slice_name, datasource_id, datasource_type, viz_type) "
+ "VALUES (:uuid, :name, :ds_id, :ds_type, :viz)"
+ )
+ for batch_start in range(0, needed, BATCH):
+ rows = [
+ {
+ "uuid": uuid_value(dialect),
+ "name": f"seed_slice_{current + i}",
+ "ds_id": table_id,
+ "ds_type": "table",
+ "viz": "table",
+ }
+ for i in range(batch_start, min(batch_start + BATCH, needed))
+ ]
+ conn.execute(sql, rows)
+ logger.info(" slices: inserted %d / %d", batch_start + len(rows), needed)
+
+
+def seed_users(conn: Connection, target: int, dry_run: bool) -> None:
+ current = count_rows(conn, "ab_user")
+ if current >= target:
+ logger.info("ab_user: %d rows (target %d) — no insert needed", current, target)
+ return
+ needed = target - current
+ logger.info("ab_user: %d → %d (+%d)", current, target, needed)
+ if dry_run:
+ return
+
+ sql = sa.text(
+ "INSERT INTO ab_user (first_name, last_name, username, email, active) "
+ "VALUES (:first, :last, :username, :email, :active)"
+ )
+ for batch_start in range(0, needed, BATCH):
+ rows = [
+ {
+ "first": "seed",
+ "last": f"user_{current + i}",
+ "username": f"seed_user_{current + i}_{uuid4().hex[:8]}",
+ "email": f"seed_user_{current + i}_{uuid4().hex[:8]}@example.invalid",
+ "active": True,
+ }
+ for i in range(batch_start, min(batch_start + BATCH, needed))
+ ]
+ conn.execute(sql, rows)
+ logger.info(" ab_user: inserted %d / %d", batch_start + len(rows), needed)
+
+
+def seed_roles(conn: Connection, target: int, dry_run: bool) -> None:
+ current = count_rows(conn, "ab_role")
+ if current >= target:
+ logger.info("ab_role: %d rows (target %d) — no insert needed", current, target)
+ return
+ needed = target - current
+ logger.info("ab_role: %d → %d (+%d)", current, target, needed)
+ if dry_run:
+ return
+
+ sql = sa.text("INSERT INTO ab_role (name) VALUES (:name)")
+ for batch_start in range(0, needed, BATCH):
+ rows = [
+ {"name": f"seed_role_{current + i}_{uuid4().hex[:8]}"}
+ for i in range(batch_start, min(batch_start + BATCH, needed))
+ ]
+ conn.execute(sql, rows)
+ logger.info(" ab_role: inserted %d / %d", batch_start + len(rows), needed)
+
+
+# ----------------------------------------------------------------------
+# Junction seeder
+# ----------------------------------------------------------------------
+
+
+def _load_existing_pairs(
+ conn: Connection, junction: str, fk1_col: str, fk2_col: str
+) -> set[tuple[int, int]]:
+ """Load existing ``(fk1, fk2)`` pairs from a junction table into a set.
+
+ Used so the seeder can skip them when generating new pairs (junction
+ tables enforce uniqueness on the FK pair). Memory is ~32 bytes/tuple
+ on CPython, so 10M existing pairs is ~320MB — acceptable for a dev
+ machine. The junction / column names come from ``JUNCTIONS``, not
+ user input, so the f-string interpolation is safe.
+ """
+ sql_text = f"SELECT {fk1_col}, {fk2_col} FROM {junction}" # noqa: S608
+ return {(row[0], row[1]) for row in conn.execute(sa.text(sql_text))}
+
+
+def _generate_new_pairs(
+ p1_ids: list[int],
+ p2_ids: list[int],
+ existing_pairs: set[tuple[int, int]],
+) -> Iterator[tuple[int, int]]:
+ """Yield ``(fk1, fk2)`` pairs from the parent1 × parent2 cross-product
+ that are not already in ``existing_pairs``."""
+ for fk1 in p1_ids:
+ for fk2 in p2_ids:
+ if (fk1, fk2) not in existing_pairs:
+ yield (fk1, fk2)
+
+
+def seed_junction(
+ conn: Connection,
+ junction: str,
+ fk1_col: str,
+ fk2_col: str,
+ parent1: str,
+ parent2: str,
+ target: int,
+ dry_run: bool,
+) -> None:
+ """Bulk-insert junction rows up to ``target`` rows total.
+
+ Generates ``(fk1, fk2)`` pairs by walking the cross-product of
+ parent1 IDs × parent2 IDs in row-major order, skipping pairs that
+ already exist. Walking the cross-product deterministically keeps
+ the script replayable: re-running with the same target is a no-op,
+ and re-running with a higher target appends new pairs in a stable
+ order regardless of how many runs preceded.
+ """
+ current = count_rows(conn, junction)
+ if current >= target:
+ logger.info(
+ "%s: %d rows (target %d) — no insert needed", junction, current, target
+ )
+ return
+ needed = target - current
+ logger.info("%s: %d → %d (+%d)", junction, current, target, needed)
+ if dry_run:
+ return
+
+ p1_ids = existing_ids(conn, parent1)
+ p2_ids = existing_ids(conn, parent2)
+ max_pairs = len(p1_ids) * len(p2_ids)
+ if max_pairs < target:
+ sys.exit(
+ f"Cannot reach {target} rows in {junction}: "
+ f"only {max_pairs} unique pairs available "
+ f"({len(p1_ids)} × {len(p2_ids)}). "
+ f"Increase parent targets and rerun."
+ )
+
+ existing_pairs: set[tuple[int, int]] = (
+ _load_existing_pairs(conn, junction, fk1_col, fk2_col) if current > 0 else set()
+ )
+ if existing_pairs:
+ logger.info(
+ " %s: loaded %d existing pairs into avoidance set",
+ junction,
+ len(existing_pairs),
+ )
+
+ insert_sql = sa.text(
+ f"INSERT INTO {junction} ({fk1_col}, {fk2_col}) " # noqa: S608
+ f"VALUES (:fk1, :fk2)"
+ )
+
+ inserted = 0
+ batch: list[dict[str, int]] = []
+ for fk1, fk2 in _generate_new_pairs(p1_ids, p2_ids, existing_pairs):
+ batch.append({"fk1": fk1, "fk2": fk2})
+ inserted += 1
+ if len(batch) == BATCH or inserted == needed:
+ conn.execute(insert_sql, batch)
+ logger.info(" %s: inserted %d / %d", junction, inserted, needed)
+ batch = []
+ if inserted == needed:
+ return
+ if inserted < needed:
+ sys.exit(
+ f"Ran out of unique pairs at {inserted}/{needed} for {junction} "
+ f"(parents have {len(p1_ids)} × {len(p2_ids)} = {max_pairs} pairs, "
+ f"{len(existing_pairs)} already present)"
+ )
+
+
+# ----------------------------------------------------------------------
+# Orchestration
+# ----------------------------------------------------------------------
+
+
+def _compute_parent_requirements(targets: dict[str, int]) -> dict[str, int]:
+ """For each parent table, return the minimum row count needed so that
+ parent1 × parent2 ≥ target for every junction it participates in.
+
+ Allocates ceil(sqrt(target)) rows per parent, balanced across the two
+ parents of each junction. The actual junction seeder will then walk
+ the cross-product to produce the target number of unique pairs.
+ """
+ parent_req: dict[str, int] = {}
+ for junction, _, _, p1, p2 in JUNCTIONS:
+ target = targets.get(junction, 0)
+ if target == 0:
+ continue
+ sqrt_n = int(target**0.5) + 1
+ parent_req[p1] = max(parent_req.get(p1, 0), sqrt_n)
+ parent_req[p2] = max(parent_req.get(p2, 0), sqrt_n)
+ return parent_req
+
+
+def _seed_parents(conn: Connection, parent_req: dict[str, int], dry_run: bool) -> None:
+ """Seed parent tables in dependency order:
+ independent parents (ab_user, ab_role) first, then dashboards / slices /
+ tables (which transitively depend on dbs, seeded inside seed_tables)."""
+ if "ab_user" in parent_req:
+ seed_users(conn, parent_req["ab_user"], dry_run)
+ if "ab_role" in parent_req:
+ seed_roles(conn, parent_req["ab_role"], dry_run)
+ if "dashboards" in parent_req:
+ seed_dashboards(conn, parent_req["dashboards"], dry_run)
+ if "slices" in parent_req:
+ seed_slices(conn, parent_req["slices"], dry_run)
+ if "tables" in parent_req:
+ seed_tables(conn, parent_req["tables"], dry_run)
+
+
+def _seed_all_junctions(
+ conn: Connection, targets: dict[str, int], dry_run: bool
+) -> None:
+ for junction, fk1, fk2, p1, p2 in JUNCTIONS:
+ target = targets.get(junction, 0)
+ if target == 0:
+ continue
+ with time_phase(f"junction:{junction}"):
+ seed_junction(conn, junction, fk1, fk2, p1, p2, target, dry_run)
+
+
+def inject_duplicates(
+ conn: Connection,
+ junction: str,
+ fk1_col: str,
+ fk2_col: str,
+ pct: float,
+ dry_run: bool,
+) -> None:
+ """Insert duplicate ``(fk1, fk2)`` rows on a non-UNIQUE junction table.
+
+ Used to stress-test the migration's ``_dedupe_by_min_id`` phase, which
+ is otherwise a no-op on cleanly-seeded data. Computes ``count =
+ current_rows * pct / 100`` and inserts that many rows by re-sampling
+ existing ``(fk1, fk2)`` pairs in row-major order. The synthetic
+ duplicates land on top of distinct existing pairs (one duplicate per
+ distinct pair, then wraps), so the migration's dedupe finds and
+ deletes them.
+
+ **Pre-condition: the table must NOT have UNIQUE on (fk1, fk2)**, i.e.,
+ the schema must be the pre-migration shape (after running
+ ``superset db downgrade``). On the post-migration schema the composite
+ PK rejects duplicates and this function will error.
+ """
+ if pct == 0:
+ return
+ current = count_rows(conn, junction)
+ count = int(current * pct / 100)
+ if count == 0:
+ logger.info(
+ "%s: 0 duplicates to inject (current=%d, pct=%g)",
+ junction,
+ current,
+ pct,
+ )
+ return
+ logger.info(
+ "%s: injecting %d duplicate rows (%g%% of %d existing)",
+ junction,
+ count,
+ pct,
+ current,
+ )
+ if dry_run:
+ return
+
+ select_sql = sa.text(
+ f"SELECT {fk1_col}, {fk2_col} FROM {junction} ORDER BY id LIMIT :n" # noqa: S608
+ )
+ sample = conn.execute(select_sql, {"n": count}).fetchall()
+ if not sample:
+ logger.warning("%s: no rows to duplicate (table is empty)", junction)
+ return
+
+ insert_sql = sa.text(
+ f"INSERT INTO {junction} ({fk1_col}, {fk2_col}) " # noqa: S608
+ f"VALUES (:fk1, :fk2)"
+ )
+ inserted = 0
+ while inserted < count:
+ batch: list[dict[str, int]] = []
+ while len(batch) < BATCH and inserted < count:
+ row = sample[inserted % len(sample)]
+ batch.append({"fk1": row[0], "fk2": row[1]})
+ inserted += 1
+ conn.execute(insert_sql, batch)
+ logger.info(" %s: injected %d / %d duplicates", junction, inserted, count)
+
+
+def _inject_dirty_data(conn: Connection, dirty_pct: float, dry_run: bool) -> None:
+ """Inject duplicate rows on every non-UNIQUE seeded junction.
+
+ The two tables that originally carried ``UNIQUE(fk1, fk2)`` are
+ skipped because their composite-PK successor (and their pre-migration
+ UNIQUE constraint) both reject duplicate inserts.
+ """
+ if dirty_pct == 0:
+ return
+ for junction, fk1, fk2, _, _ in JUNCTIONS:
+ if junction in JUNCTIONS_WITH_UNIQUE:
+ logger.info(
+ "%s: skipping duplicate injection (table has UNIQUE on FK pair)",
+ junction,
+ )
+ continue
+ with time_phase(f"dirty:{junction}"):
+ inject_duplicates(conn, junction, fk1, fk2, dirty_pct, dry_run)
+
+
+def run(targets: dict[str, int], dry_run: bool, dirty_duplicates_pct: float) -> None:
+ engine = build_engine()
+ with engine.begin() as conn:
+ parent_req = _compute_parent_requirements(targets)
+ logger.info("Required parent row counts: %s", parent_req)
+
+ with time_phase("parents"):
+ _seed_parents(conn, parent_req, dry_run)
+
+ with time_phase("junctions"):
+ _seed_all_junctions(conn, targets, dry_run)
+
+ if dirty_duplicates_pct > 0:
+ with time_phase("dirty-duplicates"):
+ _inject_dirty_data(conn, dirty_duplicates_pct, dry_run)
+
+
+# ----------------------------------------------------------------------
+# CLI
+# ----------------------------------------------------------------------
+
+
+def main() -> None:
+ parser = argparse.ArgumentParser(
+ description=__doc__,
+ formatter_class=argparse.RawDescriptionHelpFormatter,
+ )
+ for table, default in DEFAULTS.items():
+ parser.add_argument(
+ f"--{table.replace('_', '-')}",
+ type=int,
+ default=default,
+ help=f"target row count for {table} (default: {default:,})",
+ )
+ parser.add_argument(
+ "--dry-run",
+ "-n",
+ action="store_true",
+ help="print planned inserts without writing to the DB",
+ )
+ parser.add_argument(
+ "--dirty-duplicates-pct",
+ type=float,
+ default=0,
+ help=(
+ "after seeding distinct pairs, inject this percentage of duplicate "
+ "rows on each non-UNIQUE junction (slice_user, dashboard_user, "
+ "dashboard_roles). Stress-tests the migration's _dedupe_by_min_id "
+ "phase. Requires 2bee73611e32 to NOT be applied: un-apply it by "
+ "downgrading to its parent (`superset db downgrade <down_revision>`, "
+ "where <down_revision> is read from the 2bee73611e32 migration "
+ "file) — the post-migration composite PK rejects duplicates and "
+ "this will error. Default: 0 (no duplicates)."
+ ),
+ )
+ parser.add_argument(
+ "--verbose",
+ "-v",
+ action="store_true",
+ help="increase log verbosity",
+ )
+ args = parser.parse_args()
+
+ logging.basicConfig(
+ level=logging.DEBUG if args.verbose else logging.INFO,
+ format="%(asctime)s [%(levelname)s] %(message)s",
+ datefmt="%H:%M:%S",
+ )
+
+ targets = {table: getattr(args, table) for table in DEFAULTS}
Review Comment:
Addressed by subsequent commits; the flagged code is no longer current after the rebase onto master. Resolving as outdated.
##########
superset/charts/api.py:
##########
@@ -437,9 +496,29 @@ def put(self, pk: int) -> Response:
# This validates custom Schema with custom validations
except ValidationError as error:
return self.response_400(message=error.messages)
+
+ # Live version identifiers before the update (empty + query-free when
+ # ``ENABLE_VERSIONING_CAPTURE`` is off, so this stays inert under the
+ # kill-switch).
+ old_info = current_entity_version_info(Slice, pk)
Review Comment:
Addressed by subsequent commits; the flagged code is no longer current after the rebase onto master. Resolving as outdated.
##########
superset/charts/api.py:
##########
@@ -437,9 +496,29 @@ def put(self, pk: int) -> Response:
# This validates custom Schema with custom validations
except ValidationError as error:
return self.response_400(message=error.messages)
+
+ # Live version identifiers before the update (empty + query-free when
+ # ``ENABLE_VERSIONING_CAPTURE`` is off, so this stays inert under the
+ # kill-switch).
+ old_info = current_entity_version_info(Slice, pk)
+
try:
changed_model = UpdateChartCommand(pk, item).run()
- response = self.response(200, id=changed_model.id, result=item)
+ new_info = current_entity_version_info(
+ Slice, changed_model.id, changed_model.uuid
+ )
+ response = self.response(
Review Comment:
Addressed by subsequent commits; the flagged code is no longer current after the rebase onto master. Resolving as outdated.
##########
superset/commands/dashboard/update.py:
##########
@@ -59,23 +59,31 @@ def __init__(self, model_id: int, data: dict[str, Any]):
def run(self) -> Model:
self.validate()
assert self._model is not None
- self.process_tab_diff()
- self.process_native_filter_diff()
-
- # Update tags
- if (tags := self._properties.pop("tags", None)) is not None:
- update_tags(ObjectType.dashboard, self._model.id, self._model.tags, tags)
-
- # Re-serialize position_json to escape 4-byte Unicode characters
- if position_json := self._properties.get("position_json"):
- self._properties["position_json"] = json.dumps(json.loads(position_json))
-
- dashboard = DashboardDAO.update(self._model, self._properties)
- if self._properties.get("json_metadata"):
- DashboardDAO.set_dash_metadata(
- dashboard,
- data=json.loads(self._properties.get("json_metadata", "{}")),
- )
+ # Suppress autoflush during the update body so that Continuum's
+ # before_flush baseline listener does not fire mid-operation while
+ # the session is only partially populated.
+ with db.session.no_autoflush:
+ self.process_tab_diff()
+ self.process_native_filter_diff()
+
+ # Update tags
+ if (tags := self._properties.pop("tags", None)) is not None:
+ update_tags(
+ ObjectType.dashboard, self._model.id, self._model.tags, tags
+ )
+
+ # Re-serialize position_json to escape 4-byte Unicode characters
+ if position_json := self._properties.get("position_json"):
+ self._properties["position_json"] = json.dumps(
+ json.loads(position_json)
+ )
+
+ dashboard = DashboardDAO.update(self._model, self._properties)
Review Comment:
Addressed by subsequent commits; the flagged code is no longer current after the rebase onto master. Resolving as outdated.
##########
superset/dashboards/api.py:
##########
@@ -828,17 +887,32 @@ def put(self, pk: int) -> Response:
# This validates custom Schema with custom validations
except ValidationError as error:
return self.response_400(message=error.messages)
+
+ # Live version identifiers before the update (empty + query-free when
+ # ``ENABLE_VERSIONING_CAPTURE`` is off).
+ old_info = current_entity_version_info(Dashboard, pk)
Review Comment:
Addressed by subsequent commits; the flagged code is no longer current after the rebase onto master. Resolving as outdated.
##########
superset/dashboards/api.py:
##########
@@ -828,17 +887,32 @@ def put(self, pk: int) -> Response:
# This validates custom Schema with custom validations
except ValidationError as error:
return self.response_400(message=error.messages)
+
+ # Live version identifiers before the update (empty + query-free when
+ # ``ENABLE_VERSIONING_CAPTURE`` is off).
+ old_info = current_entity_version_info(Dashboard, pk)
+
try:
changed_model = UpdateDashboardCommand(pk, item).run()
last_modified_time = changed_model.changed_on.replace(
microsecond=0
).timestamp()
+ new_info = current_entity_version_info(
+ Dashboard, changed_model.id, changed_model.uuid
+ )
Review Comment:
Addressed by subsequent commits; the flagged code is no longer current after the rebase onto master. Resolving as outdated.
##########
superset/migrations/versions/2026-05-01_23-36_2bee73611e32_composite_pk_association_tables.py:
##########
@@ -0,0 +1,580 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""composite_pk_association_tables
+
+Replace the unused synthetic ``id INTEGER PRIMARY KEY`` on eight many-to-many
+association tables with a composite primary key on the two FK columns. Drops
+the now-redundant ``UniqueConstraint(fk1, fk2)`` on the two tables that
+already carry one. Pre-flight: deletes rows with NULL FK values (six tables
+allow them today) and any duplicate ``(fk1, fk2)`` rows.
+
+Motivated by SQLAlchemy-Continuum issue #129 (M2M restore against junction
+tables with surrogate PKs); also closes the data-integrity hole where six
+of the eight tables lacked DB-level uniqueness.
+
+Revision ID: 2bee73611e32
+Revises: 78a40c08b4be
+Create Date: 2026-05-01 23:36:34.050058
+
+"""
+
+import logging
+from typing import NamedTuple
+
+import sqlalchemy as sa
+from alembic import op
+from alembic.operations.base import BatchOperations
+from sqlalchemy import inspect
+from sqlalchemy.engine import Connection
+
+# revision identifiers, used by Alembic.
+revision = "2bee73611e32"
+down_revision = "78a40c08b4be"
Review Comment:
Addressed by subsequent commits; the flagged code is no longer current after the rebase onto master. Resolving as outdated.
##########
superset/migrations/versions/2026-05-01_23-36_2bee73611e32_composite_pk_association_tables.py:
##########
@@ -0,0 +1,580 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""composite_pk_association_tables
+
+Replace the unused synthetic ``id INTEGER PRIMARY KEY`` on eight many-to-many
+association tables with a composite primary key on the two FK columns. Drops
+the now-redundant ``UniqueConstraint(fk1, fk2)`` on the two tables that
+already carry one. Pre-flight: deletes rows with NULL FK values (six tables
+allow them today) and any duplicate ``(fk1, fk2)`` rows.
+
+Motivated by SQLAlchemy-Continuum issue #129 (M2M restore against junction
+tables with surrogate PKs); also closes the data-integrity hole where six
+of the eight tables lacked DB-level uniqueness.
+
+Revision ID: 2bee73611e32
+Revises: 78a40c08b4be
+Create Date: 2026-05-01 23:36:34.050058
+
+"""
+
+import logging
+from typing import NamedTuple
+
+import sqlalchemy as sa
+from alembic import op
+from alembic.operations.base import BatchOperations
+from sqlalchemy import inspect
+from sqlalchemy.engine import Connection
+
+# revision identifiers, used by Alembic.
+revision = "2bee73611e32"
+down_revision = "78a40c08b4be"
+
+logger = logging.getLogger("alembic.env")
+
+
+class AssociationTable(NamedTuple):
+ """A junction table being converted from surrogate-id PK to composite-FK PK."""
+
+ name: str
+ fk1: str
+ fk2: str
+
+
+# Order is alphabetical by table name; deterministic for review and bisection.
+AFFECTED_TABLES: list[AssociationTable] = [
+ AssociationTable("dashboard_roles", "dashboard_id", "role_id"),
+ AssociationTable("dashboard_slices", "dashboard_id", "slice_id"),
+ AssociationTable("dashboard_user", "user_id", "dashboard_id"),
+ AssociationTable("report_schedule_user", "user_id", "report_schedule_id"),
+ AssociationTable("rls_filter_roles", "role_id", "rls_filter_id"),
+ AssociationTable("rls_filter_tables", "table_id", "rls_filter_id"),
+ AssociationTable("slice_user", "user_id", "slice_id"),
+ AssociationTable("sqlatable_user", "user_id", "table_id"),
+]
+
+# These two tables already declare ``UniqueConstraint(fk1, fk2)`` in the model;
+# the composite PK subsumes it, so the migration drops the redundant constraint.
+TABLES_WITH_PRE_EXISTING_UNIQUE: set[str] = {
+ "dashboard_slices",
+ "report_schedule_user",
+}
+
+# Documentation set: tables whose FK columns are nullable in their original
+# create_table migrations (``dashboard_roles.dashboard_id`` from revision
+# e11ccdd12658 is the most recent addition). ``report_schedule_user`` is the
+# only affected table created with both FK columns ``NOT NULL`` and is
+# intentionally absent here. This set is no longer consulted at runtime — the
+# upgrade now runs the NULL-FK cleanup on every affected table because the
+# DELETE is a cheap no-op when the columns are already NOT NULL, and that
+# eliminates the risk of bugs from this set going stale (the
+# ``dashboard_roles`` omission caught in PR review was exactly that bug).
+TABLES_WITH_NULLABLE_FKS: set[str] = {
+ "dashboard_roles",
+ "dashboard_slices",
+ "dashboard_user",
+ "rls_filter_roles",
+ "rls_filter_tables",
+ "slice_user",
+ "sqlatable_user",
+}
+
+
+def _check_no_external_fks_to_id(conn: Connection) -> None:
+ """Raise ``RuntimeError`` if any foreign key in the database references one
+ of the eight junction-table ``id`` columns. Uses SQLAlchemy's ``Inspector``
+ for dialect-agnostic introspection across PostgreSQL, MySQL, and SQLite.
+
+ Scope limitation: ``Inspector.get_table_names()`` returns tables in the
+ connection's default schema only. On PostgreSQL deployments where Superset
+ metadata lives in a non-default schema, or on multi-schema deployments
+ that allow cross-schema FKs, an external FK in another schema would not
+ be detected. This is acceptable for the standard single-schema
+ deployment that Superset documents; operators with multi-schema
+ metadata should run the equivalent inventory query against
+ ``information_schema.referential_constraints`` themselves before
+ applying.
+ """
+ affected = {t.name for t in AFFECTED_TABLES}
+ insp = inspect(conn)
+ for table_name in insp.get_table_names():
+ if table_name in affected:
+ continue
+ for fk in insp.get_foreign_keys(table_name):
+ if fk["referred_table"] in affected and "id" in fk["referred_columns"]:
+ raise RuntimeError(
+ f"Cannot drop synthetic id from {fk['referred_table']}: "
+ f"external FK {fk.get('name', '<unnamed>')} on {table_name} "
+ f"references {fk['referred_table']}({fk['referred_columns']}). "
+ "Drop or migrate the referencing FK before applying this "
+ "migration."
+ )
+
+
+def _table_clause(t: AssociationTable) -> sa.sql.expression.TableClause:
+ """Build a lightweight SQLAlchemy ``TableClause`` for ``t`` exposing the
+ columns the helper queries reference (``id``, ``fk1``, ``fk2``). Used so
+ that the dedupe / cleanup / assert SQL can be expressed via SQLAlchemy
+ core constructs rather than via string interpolation."""
+ return sa.table(t.name, sa.column("id"), sa.column(t.fk1), sa.column(t.fk2))
+
+
+def _delete_null_fk_rows(conn: Connection, t: AssociationTable) -> int:
+ """Delete rows where ``t.fk1`` or ``t.fk2`` is NULL on ``t.name``.
+
+ Returns the deletion count. Required because primary-key columns must be
+ NOT NULL; the PK-add downstream would fail with a cryptic constraint
+ violation if any NULL-FK rows survived. Run unconditionally on every
+ affected table — see ``TABLES_WITH_NULLABLE_FKS`` above for the rationale.
+ """
+ tbl = _table_clause(t)
+ stmt = sa.delete(tbl).where(sa.or_(tbl.c[t.fk1].is_(None), tbl.c[t.fk2].is_(None)))
+ result = conn.execute(stmt)
+ n = result.rowcount or 0
+ if n:
+ logger.warning(
+ "Deleted %d row(s) with NULL FK from %s before composite-PK promotion",
+ n,
+ t.name,
+ )
+ return n
+
+
+def _dedupe_by_min_id(conn: Connection, t: AssociationTable) -> int:
+ """Delete duplicate ``(t.fk1, t.fk2)`` rows from ``t.name`` keeping ``MIN(id)``.
+
+ Returns the deletion count. The ``NOT IN`` argument is wrapped in an
+ extra ``SELECT keep_id FROM (...) AS s`` derived table because MySQL
+ rejects ``DELETE FROM t WHERE id NOT IN (SELECT MIN(id) FROM t GROUP BY
+ ...)`` with ERROR 1093 unless the inner SELECT is materialized through
+ a derived table. SQLAlchemy's ``.subquery()`` produces that wrap.
+
+ Logs a sample (up to 10) of the discarded ``(fk1, fk2, id)`` tuples at
+ WARN before deletion, so operators can audit which rows are dropped —
+ the "keep ``MIN(id)``" policy preserves the original row, which is
+ correct in practice but discards any later, semantically-identical
+ re-grants.
+ """
+ tbl = _table_clause(t)
+
+ keep_min = (
+ sa.select(sa.func.min(tbl.c.id).label("keep_id"))
+ .group_by(tbl.c[t.fk1], tbl.c[t.fk2])
+ .subquery("keep_min")
+ )
+ keep_ids = sa.select(keep_min.c.keep_id)
+ discarded = tbl.c.id.notin_(keep_ids)
+
+ sample_stmt = (
+ sa.select(tbl.c[t.fk1], tbl.c[t.fk2], tbl.c.id).where(discarded).limit(10)
+ )
+ sample = list(conn.execute(sample_stmt))
+
+ delete_stmt = sa.delete(tbl).where(discarded)
+ result = conn.execute(delete_stmt)
+ n = result.rowcount or 0
+ if n:
+ logger.warning(
+ "Deduped %d duplicate row(s) from %s; sample of discarded "
+ "(%s, %s, id) tuples (up to 10): %s",
+ n,
+ t.name,
+ t.fk1,
+ t.fk2,
+ sample,
+ )
+ return n
+
+
+def _assert_no_duplicates(conn: Connection, t: AssociationTable) -> None:
+ """Raise ``RuntimeError`` if any ``(t.fk1, t.fk2)`` duplicate group remains.
+
+ Called after ``_dedupe_by_min_id`` to surface silent dialect-dependent
+ dedupe failures (e.g., a MySQL syntax issue) as an actionable error
+ before the PK-add fires with a less-helpful constraint-violation message.
+ """
+ tbl = _table_clause(t)
+ duplicate_groups = (
+ sa.select(sa.literal(1))
+ .select_from(tbl)
+ .group_by(tbl.c[t.fk1], tbl.c[t.fk2])
+ .having(sa.func.count() > 1)
+ .subquery("duplicate_groups")
+ )
+ count_stmt = sa.select(sa.func.count()).select_from(duplicate_groups)
+ if remaining := conn.scalar(count_stmt) or 0:
+ raise RuntimeError(
+ f"Dedupe failed for {t.name}: {remaining} duplicate "
+ f"({t.fk1}, {t.fk2}) groups remain after _dedupe_by_min_id. "
+ f"Check the dedupe SQL for dialect {conn.dialect.name}."
+ )
+
+
+def _build_pre_upgrade_table(
+ insp: sa.engine.reflection.Inspector,
+ t: AssociationTable,
+ fks: list[dict] | None = None,
+) -> sa.Table:
+ """Build a ``Table`` object representing the pre-upgrade schema of ``t``,
+ explicitly *without* any redundant ``UniqueConstraint(t.fk1, t.fk2)``.
+ Used as ``copy_from`` to ``batch_alter_table`` so the rebuilt table
+ omits the unnamed UNIQUE constraint deterministically across dialects
+ (SQLite reflects unnamed UNIQUEs with ``name=None``, defeating the
+ standard ``batch_op.drop_constraint(name)`` path).
+
+ Reflects column types and FK targets (with original FK constraint names
+ preserved) from the live database; only the redundant UNIQUE is omitted.
+
+ *fks* lets a caller pass a pre-captured ``get_foreign_keys`` result.
+ The MySQL upgrade path drops the live FK constraints before building
+ this table, so re-reflecting here would only see them via the
+ Inspector's per-instance ``info_cache`` — an implementation detail,
+ not a contract. Passing the pre-drop list makes the dependency
+ explicit instead of relying on reflection caching.
+ """
+ md = sa.MetaData()
+ if fks is None:
+ fks = insp.get_foreign_keys(t.name)
+ fks_for_col: dict[str, list[dict]] = {}
+ for fk in fks:
+ for col_name in fk["constrained_columns"]:
+ fks_for_col.setdefault(col_name, []).append(fk)
+
+ cols: list[sa.Column] = []
+ for c in insp.get_columns(t.name):
+ col_kwargs = {"nullable": c.get("nullable", True)}
+ if c["name"] == "id":
+ col_kwargs["primary_key"] = True
+ col_kwargs["autoincrement"] = True
+ fk_args = []
+ for fk in fks_for_col.get(c["name"], []):
+ idx = fk["constrained_columns"].index(c["name"])
+ target = f"{fk['referred_table']}.{fk['referred_columns'][idx]}"
+ options = {}
+ if fk.get("options", {}).get("ondelete"):
+ options["ondelete"] = fk["options"]["ondelete"]
+ if fk.get("name"):
+ options["name"] = fk["name"]
+ fk_args.append(sa.ForeignKey(target, **options))
+ cols.append(sa.Column(c["name"], c["type"], *fk_args, **col_kwargs))
+ return sa.Table(t.name, md, *cols)
+
+
+def _drop_redundant_unique_by_name(
+ conn: Connection, insp: sa.engine.reflection.Inspector, t: AssociationTable
+) -> None:
+ """Drop the redundant ``UNIQUE(fk1, fk2)`` constraint by its reflected
+ name on PostgreSQL / MySQL.
+
+ The two tables in ``TABLES_WITH_PRE_EXISTING_UNIQUE`` carry a UNIQUE
+ constraint that the composite primary key subsumes. PostgreSQL and
+ MySQL both auto-name UNIQUE constraints (``<table>_<cols>_key`` on
+ Postgres, ``<table>_<col>_<n>`` or the explicit ``uq_*`` we may have
+ given it on MySQL), so they're reflectable by name. SQLite is
+ handled separately via ``recreate="always"`` + ``copy_from`` because
+ it reflects unnamed UNIQUEs with ``name=None``.
+
+ No-op if no matching UNIQUE is found (defensive — re-runs after a
+ partial application should not error).
+ """
+ for uc in insp.get_unique_constraints(t.name):
+ if set(uc.get("column_names", [])) == {t.fk1, t.fk2} and uc.get("name"):
+ op.drop_constraint(uc["name"], t.name, type_="unique")
+ return
+
+
+# MySQL ON DELETE actions that the downgrade re-create loop is allowed
+# to interpolate into raw SQL. The reflected value comes from MySQL's
+# information_schema (so not user input), but a whitelist eliminates
+# the "what if an unexpected value appears" question entirely. The
+# four entries are the SQL-standard set; SET DEFAULT is intentionally
+# excluded because InnoDB silently downgrades it to NO ACTION.
+_VALID_ONDELETE_ACTIONS: frozenset[str] = frozenset(
+ {"CASCADE", "SET NULL", "RESTRICT", "NO ACTION"}
+)
+
+
+def _enforce_not_null_for_sqlite(
+ batch_op: BatchOperations, t: AssociationTable, conn: Connection
+) -> None:
+ """Force ``NOT NULL`` on the FK columns post-PK-promotion on SQLite only.
+
+ SQLite has a long-standing quirk: composite ``PRIMARY KEY`` does not
+ promote constituent columns to ``NOT NULL`` (only ``INTEGER PRIMARY KEY``
+ does). PostgreSQL and MySQL implicitly promote the PK columns to
+ ``NOT NULL`` when the constraint is added, making the explicit
+ ``alter_column`` redundant there.
+
+ Skipping the ``alter_column`` on MySQL is also functionally required:
+ MySQL 8 rejects ``ALTER COLUMN`` on a column that participates in a
+ foreign key constraint with ``ERROR 1832 (HY000): Cannot change column
+ 'X': used in a foreign key constraint 'Y'`` whenever the table has
+ data — even when the only change is ``NULL`` → ``NOT NULL`` and the
+ column is already part of a freshly-added composite primary key (which
+ InnoDB has just made implicitly ``NOT NULL`` anyway). The error fires
+ on populated tables but not on empty ones, which is why CI's
+ ``test-mysql`` shard (fresh schema) didn't catch this and a real
+ production-shaped install does.
+
+ Only SQLite still needs the explicit step, and SQLite has no FK
+ enforcement objection.
+ """
+ if conn.dialect.name == "sqlite":
+ batch_op.alter_column(t.fk1, existing_type=sa.Integer, nullable=False)
+ batch_op.alter_column(t.fk2, existing_type=sa.Integer, nullable=False)
+
+
+def upgrade() -> None:
+ conn = op.get_bind()
+ _check_no_external_fks_to_id(conn)
+ insp = inspect(conn)
+
+ for t in AFFECTED_TABLES:
+ # Resumability guard: on MySQL every DDL statement auto-commits, so
+ # a failure at table N of 8 leaves tables 1..N-1 already converted
+ # while ``alembic_version`` is still un-stamped. Without this guard
+ # a re-run would fail at table 1 (``drop_column("id")`` on a table
+ # that no longer has ``id``), and ``downgrade`` can't run either
+ # (the revision was never stamped) — recovery would need manual
+ # surgery. A converted table is identified by the absent ``id``
+ # column; skipping it makes re-running the upgrade safe on every
+ # dialect (Postgres/SQLite wrap the migration in a transaction, so
+ # the guard is simply never hit there).
+ if "id" not in {c["name"] for c in insp.get_columns(t.name)}:
+ logger.info(
+ "%s: already converted (no surrogate id column); skipping",
+ t.name,
+ )
+ continue
+
+ # Run NULL-FK cleanup unconditionally: it is a no-op DELETE on tables
+ # whose FK columns are already NOT NULL (cheap), and skipping it on a
+ # table whose FK was nullable would leave the PK-add to fail with a
+ # cryptic constraint violation. Cf. ``TABLES_WITH_NULLABLE_FKS`` above
+ # for documentation of which tables are known to have nullable FKs.
+ _delete_null_fk_rows(conn, t)
+ _dedupe_by_min_id(conn, t)
+ _assert_no_duplicates(conn, t)
+
+ # Two tables (``dashboard_slices``, ``report_schedule_user``)
+ # carry a redundant ``UNIQUE(fk1, fk2)`` that the composite PK
+ # subsumes. Three dialect-specific paths:
+ #
+ # * **PostgreSQL** — the UNIQUE constraint has a stable
+ # reflected name (Postgres default convention), so we
+ # ``DROP CONSTRAINT`` by name and then run the structural
+ # change as direct ALTER. This avoids the full-table copy
+ # that ``recreate="always"`` would trigger
+ # (``CREATE TABLE AS SELECT → DROP → RENAME``), holding
+ # ``ACCESS EXCLUSIVE`` only for the (much shorter) PK
+ # index build instead of the full copy duration.
+ #
+ # * **MySQL** — InnoDB binds the FK constraints to the
+ # redundant UNIQUE's underlying index for back-reference,
+ # so a direct ``DROP CONSTRAINT`` of the UNIQUE raises
+ # ``ERROR 1553``. Use ``recreate="always"`` to rebuild the
+ # table without the UNIQUE; drop the FKs first to dodge
+ # the ``ERROR 1826`` (duplicate FK constraint name) that
+ # the temp-table phase would otherwise provoke. The FKs
+ # are re-created automatically as part of ``copy_from``.
+ #
+ # * **SQLite** — unnamed UNIQUE constraints reflect with
+ # ``name=None`` and can't be dropped by name. Use
+ # ``recreate="always"`` + ``copy_from`` (omits UNIQUE).
+ # SQLite always rebuilds for PK changes anyway, so the
+ # recreate isn't extra cost there.
+ if t.name in TABLES_WITH_PRE_EXISTING_UNIQUE:
+ if conn.dialect.name == "postgresql":
+ _drop_redundant_unique_by_name(conn, insp, t)
+ with op.batch_alter_table(t.name) as batch_op:
+ batch_op.drop_column("id")
+ batch_op.create_primary_key(f"pk_{t.name}", [t.fk1, t.fk2])
+ _enforce_not_null_for_sqlite(batch_op, t, conn)
+ else:
+ # Capture the FK list BEFORE dropping: the copy_from table
+ # below must embed these constraints, and re-reflecting
+ # after the drop only works via the Inspector's
+ # per-instance info_cache (see _build_pre_upgrade_table).
+ pre_drop_fks = insp.get_foreign_keys(t.name)
+ if conn.dialect.name == "mysql":
+ for fk in pre_drop_fks:
+ if fk_name := fk.get("name"):
+ op.drop_constraint(fk_name, t.name, type_="foreignkey")
Review Comment:
Addressed by subsequent commits; the flagged code is no longer current after the rebase onto master. Resolving as outdated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(versioning): cross-entity version activity view [superset]
Posted by "mikebridge (via GitHub)" <gi...@apache.org>.
mikebridge commented on code in PR #41076:
URL: https://github.com/apache/superset/pull/41076#discussion_r3574999266
##########
superset/versioning/activity/queries.py:
##########
@@ -0,0 +1,754 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""DB-touching helpers for the activity-view read path.
+
+All Phase A relationship walks (``charts_attached_to_dashboard``,
+``datasets_used_by_chart``, ``batch_datasets_used_by_charts``),
+the Phase B change-record fetch (``fetch_change_records`` /
+``_select_change_rows_for_kinds``), the name-denormalization helpers
+(``_resolve_names_for_kind`` / ``apply_entity_name_denormalization``), the
+path-entity resolution helper (``resolve_path_entity``), and the
+tombstone-state lookup (``check_entity_tombstones``) live here.
+
+Each helper is a thin SELECT-and-shape function — no orchestration,
+no business logic. Callers in :mod:`scope`, :mod:`render`, and
+:mod:`orchestrator` compose them into the end-to-end request.
+
+**Inline imports.** Continuum's ``version_class`` / ``versioning_manager``
+and the Superset model classes are imported inside each helper because
+this package is loaded from ``init_versioning()`` before all SQLAlchemy
+mappers are configured.
+"""
+
+from __future__ import annotations
+
+import logging
+from datetime import datetime
+from typing import Any
+from uuid import UUID
+
+import sqlalchemy as sa
+from flask_appbuilder import Model
+
+from superset.extensions import db
+from superset.versioning.activity.kinds import (
+ API_KIND_TO_TABLE,
+ chunked_ids,
+ EntityWindows,
+ load_live_model,
+ NAME_COLUMN,
+ NOT_FOUND_EXC,
+ TABLE_KIND_TO_API,
+ Window,
+)
+from superset.versioning.activity.windows import row_within_any_window
+from superset.versioning.changes import version_changes_table
+
+logger = logging.getLogger(__name__)
+
+# ---- Path-entity resolution -----------------------------------------------
+
+
+def resolve_path_entity(model_cls: type[Model], entity_uuid: UUID) -> tuple[Any, int]:
+ """Resolve *entity_uuid* to ``(live_entity, entity_id)`` or raise a
+ typed 404.
+
+ Soft-delete handling is inherited transparently from
+ :func:`superset.versioning.queries.find_active_by_uuid` once it
+ learns to filter out ``deleted_at IS NOT NULL`` rows; at that point
+ soft-deleted paths will also raise here.
+ """
+ # pylint: disable=import-outside-toplevel
+ from superset.versioning.queries import find_active_by_uuid
+
+ entity = find_active_by_uuid(model_cls, entity_uuid)
+ if entity is None:
+ api_kind = model_cls.__name__
+ exc_cls = NOT_FOUND_EXC.get(api_kind)
+ if exc_cls is None:
+ raise LookupError(
+ f"Activity view does not support model class {api_kind!r}"
+ )
+ raise exc_cls(str(entity_uuid))
+ return entity, entity.id
+
+
+def first_tracked_tx(
+ model_cls: type[Model], entity_id: int, entity_uuid: UUID
+) -> int | None:
+ """Return the earliest Continuum transaction_id of the shadow rows
+ that belong to *this* live entity, matched on ``(id, uuid)``; ``None``
+ when the entity has no tracked history yet.
+
+ Used to lower-bound the self-scope window. Matching on the integer
+ ``id`` alone would inherit a previously hard-deleted entity's history
+ under id reuse (SQLite/MySQL reuse ``max(id)+1``); pinning the uuid
+ too — the same discrimination :func:`mark_first_tracked_saves` uses —
+ scopes the window to the current entity's own transactions.
+ """
+ # pylint: disable=import-outside-toplevel
+ from sqlalchemy_continuum import version_class
+
+ shadow_tbl = version_class(model_cls).__table__
+ stmt = sa.select(sa.func.min(shadow_tbl.c.transaction_id)).where(
+ shadow_tbl.c.id == entity_id,
+ shadow_tbl.c.uuid == entity_uuid,
+ )
+ return db.session.connection().execute(stmt).scalar()
+
+
+# ---- Phase A: relationship-traversal queries ------------------------------
+
+
+def charts_attached_to_dashboard(dashboard_id: int) -> list[tuple[int, Window]]:
+ """Return ``(slice_id, window)`` for every chart that has ever been on
+ *dashboard_id*, with each association's validity window in
+ transaction-id space.
+
+ Reads from ``dashboard_slices_version`` (Continuum's auto-generated
+ M2M shadow). Rows with ``operation_type = 2`` (DELETE) are excluded
+ so we don't synthesize a phantom window from a detachment row.
+ """
+ # pylint: disable=import-outside-toplevel
+ from sqlalchemy_continuum import version_class
+
+ from superset.models.dashboard import Dashboard
+
+ metadata = version_class(Dashboard).__table__.metadata
+ m2m_tbl = metadata.tables.get("dashboard_slices_version")
+ if m2m_tbl is None:
+ return []
+
+ rows = (
+ db.session.connection()
+ .execute(
+ sa.select(
+ m2m_tbl.c.slice_id,
+ m2m_tbl.c.transaction_id,
+ m2m_tbl.c.end_transaction_id,
+ ).where(
+ m2m_tbl.c.dashboard_id == dashboard_id,
+ m2m_tbl.c.operation_type != 2,
+ m2m_tbl.c.slice_id.is_not(None),
+ )
+ )
+ .all()
+ )
+ result: list[tuple[int, Window]] = []
+ for row in rows:
+ try:
+ window = Window(row[1], row[2])
+ except ValueError:
+ # A degenerate shadow row (end_tx <= start_tx) must not 500 the
+ # endpoint; skip it and leave a breadcrumb for investigation.
+ logger.warning(
+ "activity: skipping degenerate dashboard_slices_version row "
+ "(dashboard_id=%s, slice_id=%s, tx=%s, end_tx=%s)",
+ dashboard_id,
+ row[0],
+ row[1],
+ row[2],
+ )
+ continue
+ result.append((row[0], window))
+ return result
+
+
+def datasets_used_by_chart(slice_id: int) -> list[tuple[int, Window]]:
+ """Return ``(datasource_id, window)`` for every dataset that *slice_id*
+ has ever pointed at, with each association's validity window.
+
+ Single-slice form, used by ``_resolve_chart_scope`` where there
+ is only one chart to walk. The dashboard-scope path calls
+ :func:`batch_datasets_used_by_charts` instead so the query fires
+ once for all slices on the dashboard, not once per slice.
+
+ Reads from ``slices_version`` (the chart parent shadow). Filters to
+ ``datasource_type = 'table'`` because the activity view only follows
+ the chart → ``SqlaTable`` dependency edge (not legacy/other
+ datasources). Rows with ``operation_type = 2`` are excluded.
+ """
+ return batch_datasets_used_by_charts({slice_id}).get(slice_id, [])
+
+
+def batch_datasets_used_by_charts(
+ slice_ids: set[int],
+) -> dict[int, list[tuple[int, Window]]]:
+ """Batch form of :func:`datasets_used_by_chart`. Returns
+ ``{slice_id: [(dataset_id, window), ...]}`` in a single query so the
+ dashboard-scope walker doesn't fire one query per chart on the
+ dashboard. The previous per-slice shape became O(n_charts) round-
+ trips, which dominated ``get_activity`` latency on dashboards with
+ rich history (profile run 2026-05-26 showed `resolve_scope`
+ accounting for ~1.9s out of 4s p95).
+ """
+ if not slice_ids:
+ return {}
+
+ # pylint: disable=import-outside-toplevel
+ from sqlalchemy_continuum import version_class
+
+ from superset.models.slice import Slice
+
+ slices_tbl = version_class(Slice).__table__
+ grouped: dict[int, list[tuple[int, Window]]] = {}
+ # Chunk the IN-clause under SQLite's bind-variable floor (a dashboard can
+ # carry more charts than the floor allows in one statement).
+ rows: list[Any] = []
+ for chunk in chunked_ids(slice_ids):
+ rows.extend(
+ db.session.connection()
+ .execute(
+ sa.select(
+ slices_tbl.c.id,
+ slices_tbl.c.datasource_id,
+ slices_tbl.c.transaction_id,
+ slices_tbl.c.end_transaction_id,
+ ).where(
+ slices_tbl.c.id.in_(chunk),
+ slices_tbl.c.datasource_type == "table",
+ slices_tbl.c.operation_type != 2,
+ slices_tbl.c.datasource_id.is_not(None),
+ )
+ )
+ .mappings()
+ .all()
+ )
+ for row in rows:
+ try:
+ window = Window(row["transaction_id"], row["end_transaction_id"])
+ except ValueError:
+ # A degenerate shadow row (end_tx <= start_tx) must not 500 the
+ # endpoint; skip it and leave a breadcrumb for investigation.
+ logger.warning(
+ "activity: skipping degenerate slices_version row "
+ "(slice_id=%s, datasource_id=%s, tx=%s, end_tx=%s)",
+ row["id"],
+ row["datasource_id"],
+ row["transaction_id"],
+ row["end_transaction_id"],
+ )
+ continue
+ grouped.setdefault(row["id"], []).append((row["datasource_id"], window))
+ return grouped
+
+
+# ---- Phase B: change-record fetch -----------------------------------------
+
+
+def fetch_change_records(
+ entity_window_tuples: list[EntityWindows],
+ since: datetime | None,
+ until: datetime | None,
+) -> tuple[list[dict[str, Any]], bool]:
+ """Fetch all ``version_changes`` rows matching any of the supplied
+ entity-window tuples, joined with ``version_transaction`` for
+ ``issued_at`` and ``user_id``.
+
+ Each tuple is ``(api_kind, entity_id, [(start_tx, end_tx), ...])``;
+ a record matches when ``entity_kind`` equals the table-stored form
+ of *api_kind*, ``entity_id`` matches, and ``transaction_id`` falls
+ inside at least one of the entity's windows. ``since``/``until``
+ further restrict by ``issued_at``.
+
+ Implementation: one SELECT per kind with ``entity_id IN (...)`` and
+ a wide ``transaction_id`` bound (the union of all windows for that
+ kind). Per-window precision is applied in Python afterward. This
+ keeps the SQL shape proportional to the number of *kinds* (≤3) and
+ the bound proportional to the union of windows, not the cross-
+ product of (entity, window) — which previously generated one OR
+ clause per (entity, window) pair and hit SQLite's
+ ``SQLITE_MAX_EXPR_DEPTH`` limit on dashboards with many slices
+ or many historical attachment windows.
+
+ The visibility filter runs after this function (records
+ the requester can't read are silently dropped and must not
+ contribute to ``count``), so the orchestrator paginates in Python
+ over the filtered list — there is no DB-level page ``OFFSET`` here.
+ There IS a per-kind safety ceiling (``_MAX_FETCHED_RECORDS``) that
+ bounds how much history a single request materializes (to at most
+ ``n_kinds * _MAX_FETCHED_RECORDS``); when a kind exhausts it, the
+ second return value is ``True``, the returned stream is clamped to a
+ clean time cut (records older than the highest per-kind fetch floor
+ are dropped so truncation is a time boundary, not per-kind holes), and
+ the caller surfaces ``truncated`` on the response.
+
+ Returns ``(records, truncated)``. Records are ordered by
+ ``(issued_at DESC, transaction_id DESC, sequence DESC)`` — the
+ secondary keys break ties for the stable-ordering contract.
+ """
+ if not entity_window_tuples:
+ return [], False
+
+ # Group windows by (table_kind, entity_id) and by table_kind for SQL
+ # narrowing. The fetch is per-kind; the post-filter is per-entity.
+ windows_by_entity: dict[tuple[str, int], list[Window]] = {}
+ ids_by_kind: dict[str, set[int]] = {}
+ for api_kind, entity_id, windows in entity_window_tuples:
+ table_kind = API_KIND_TO_TABLE.get(api_kind)
+ if table_kind is None or not windows:
+ continue
+ ids_by_kind.setdefault(table_kind, set()).add(entity_id)
+ windows_by_entity.setdefault((table_kind, entity_id), []).extend(windows)
+
+ if not ids_by_kind:
+ return [], False
+
+ # Per-kind transaction_id bounds = the union of that kind's windows.
+ # Pushing these into the SQL WHERE ensures the per-statement LIMIT
+ # selects from IN-WINDOW rows. Without it, a related entity whose
+ # association window is far in the past would have the newest ``limit``
+ # (out-of-window) rows fetched and discarded, silently dropping its
+ # in-window records that lie beyond the limit. ``end_tx = None``
+ # (open-ended/current) means no upper bound for that kind.
+ bounds_by_kind: dict[str, tuple[int, int | None]] = {}
+ for (table_kind, _entity_id), windows in windows_by_entity.items():
+ for w in windows:
+ cur = bounds_by_kind.get(table_kind)
+ if cur is None:
+ bounds_by_kind[table_kind] = (w.start_tx, w.end_tx)
+ continue
+ lo = min(cur[0], w.start_tx)
+ hi = None if (cur[1] is None or w.end_tx is None) else max(cur[1], w.end_tx)
+ bounds_by_kind[table_kind] = (lo, hi)
+
+ rows, truncated, truncation_floor = _select_change_rows_for_kinds(
+ ids_by_kind, bounds_by_kind, since, until, _MAX_FETCHED_RECORDS
+ )
+ filtered = [
+ row
+ for row in rows
+ if row_within_any_window(
+ row, windows_by_entity.get((row["entity_kind"], row["entity_id"]), [])
+ )
+ ]
+ if truncated and truncation_floor is not None:
+ # Collapse truncation to a clean time cut. Below the highest per-kind
+ # fetch floor at least one kind is missing rows, so surfacing the
+ # other kinds there would present an incomplete ("nothing else
+ # changed") picture. Drop everything older than the floor so
+ # ``truncated=True`` means "complete at/after this instant, nothing
+ # shown before it" — a time boundary rather than per-kind holes.
+ filtered = [r for r in filtered if r["issued_at"] >= truncation_floor]
+ # Sort key must be TOTAL so pagination is stable across requests: two
+ # records from different entities can share (issued_at, transaction_id,
+ # sequence), so append (entity_kind, entity_id) to break remaining ties
+ # deterministically. Without these the relative order of tied records
+ # depends on set-iteration order and a record could shift pages.
+ filtered.sort(
+ key=lambda r: (
+ r["issued_at"],
+ r["transaction_id"],
+ r["sequence"],
+ r["entity_kind"],
+ r["entity_id"],
+ ),
+ reverse=True,
+ )
+ return filtered, truncated
+
+
+def _select_change_rows_for_kinds(
+ ids_by_kind: dict[str, set[int]],
+ bounds_by_kind: dict[str, tuple[int, int | None]],
+ since: datetime | None,
+ until: datetime | None,
+ limit: int,
+) -> tuple[list[dict[str, Any]], bool, datetime | None]:
+ """Fire one SELECT per entity_kind with ``entity_id IN (...)``;
+ concatenate the results. Each SELECT joins ``version_transaction``
+ + ``ab_user`` so the orchestrator has the columns it needs for
+ decoration.
+
+ Per-kind, not one query: SQLAlchemy's ``tuple_(entity_kind,
+ entity_id).in_(...)`` would collapse the three queries into one,
+ but its SQL emission is not portable across Postgres, MySQL, and
+ SQLite. The per-kind shape is the correct trade-off given
+ Superset's multi-dialect requirement (at most 3 round-trips per
+ request, bounded by the kind taxonomy). Do not "optimise" into a
+ composite-tuple IN clause without verifying the SQL on all three
+ dialects.
+
+ **Init-order dependency.** ``tx_tbl.c.action_kind`` resolves only
+ after ``init_versioning()`` has run — the column is appended onto
+ Continuum's transaction Table by
+ ``superset.versioning.factory.VersionTransactionFactory`` at app
+ start via ``append_column`` + ``add_property``. This helper is
+ safe to call from request-path code because the app is fully
+ initialised by then; calling it from a script that imports the
+ versioning package without going through ``init_versioning()``
+ will raise ``AttributeError`` on the ``action_kind`` attribute
+ access below."""
+ # pylint: disable=import-outside-toplevel
+ from sqlalchemy_continuum import versioning_manager
+
+ from superset import security_manager
+
+ tx_tbl = versioning_manager.transaction_cls.__table__
+ user_tbl = security_manager.user_model.__table__
+ vc = version_changes_table
+ join_tree = vc.join(tx_tbl, vc.c.transaction_id == tx_tbl.c.id).outerjoin(
+ user_tbl, tx_tbl.c.user_id == user_tbl.c.id
+ )
+ select_cols = (
+ vc.c.transaction_id,
+ vc.c.entity_kind,
+ vc.c.entity_id,
+ vc.c.sequence,
+ vc.c.kind,
+ vc.c.operation,
+ vc.c.path,
+ vc.c.from_value,
+ vc.c.to_value,
+ tx_tbl.c.issued_at,
+ tx_tbl.c.user_id,
+ # ``action_kind`` is the high-level avenue (restore / import /
+ # clone / NULL=ordinary save) stamped by the originating
+ # command via the change-record listener. All records sharing a
+ # ``transaction_id`` share the same value. The column is
+ # declared on the Continuum Table by ``VersionTransactionFactory``,
+ # so ``tx_tbl.c.action_kind`` resolves cleanly here. See
+ # the three change-record dimensions.
+ tx_tbl.c.action_kind,
+ user_tbl.c.id.label("changed_by_id"),
+ user_tbl.c.first_name,
+ user_tbl.c.last_name,
+ )
+
+ out: list[dict[str, Any]] = []
+ truncated = False
+ truncation_floors: list[datetime] = []
+ for table_kind, entity_ids in ids_by_kind.items():
+ # The fetch budget is per KIND, not per id-chunk. A kind whose
+ # entity_ids span several bind-variable chunks shares one ``limit``
+ # across those chunks, so the rows a single request materializes are
+ # bounded by ``n_kinds * limit`` (<= 3 * _MAX_FETCHED_RECORDS), not
+ # ``n_chunks * limit``. ``entity_ids`` is chunked to stay inside
+ # SQLite's ``SQLITE_MAX_VARIABLE_NUMBER`` floor (default 999 in many
+ # builds); Postgres + MySQL accept the full list but the chunk is
+ # dialect-agnostic for simplicity.
+ kind_rows: list[dict[str, Any]] = []
+ for chunk in chunked_ids(entity_ids):
+ remaining = limit - len(kind_rows)
+ if remaining <= 0:
+ break
+ stmt = (
+ sa.select(*select_cols)
+ .select_from(join_tree)
+ .where(
+ vc.c.entity_kind == table_kind,
+ vc.c.entity_id.in_(chunk),
+ )
+ )
+ # Bound by the kind's window union so the LIMIT picks in-window
+ # rows (see fetch_change_records). The per-entity window filter
+ # still runs in Python afterwards for exact membership.
+ tx_lo, tx_hi = bounds_by_kind[table_kind]
+ stmt = stmt.where(vc.c.transaction_id >= tx_lo)
+ if tx_hi is not None:
+ stmt = stmt.where(vc.c.transaction_id < tx_hi)
+ if since is not None:
+ stmt = stmt.where(tx_tbl.c.issued_at >= since)
+ if until is not None:
+ stmt = stmt.where(tx_tbl.c.issued_at < until)
+ # Match fetch_change_records' final Python sort key order
+ # (issued_at, transaction_id, sequence, entity_id) so the
+ # per-kind budget keeps exactly the rows the final sort ranks
+ # highest. entity_kind is constant within a per-kind statement.
+ stmt = stmt.order_by(
+ tx_tbl.c.issued_at.desc(),
+ vc.c.transaction_id.desc(),
+ vc.c.sequence.desc(),
+ vc.c.entity_id.desc(),
+ ).limit(remaining)
+ kind_rows.extend(
+ dict(row)
+ for row in db.session.connection().execute(stmt).mappings().all()
+ )
+ if len(kind_rows) >= limit:
+ # This kind exhausted its budget; older rows almost certainly
+ # exist beyond it. Record the oldest ``issued_at`` fetched for
+ # the kind so the caller can collapse truncation into a single
+ # clean time cut across all kinds (see fetch_change_records).
+ truncated = True
+ truncation_floors.append(min(r["issued_at"] for r in kind_rows))
+ out.extend(kind_rows)
+ truncation_floor = max(truncation_floors) if truncation_floors else None
+ return out, truncated, truncation_floor
+
+
+# Per-KIND safety ceiling on how many change rows a single activity request
+# will materialize for one entity kind (shared across that kind's id-chunks).
+# Bounds memory/CPU for a path entity with very long history or many related
+# entities to <= n_kinds * _MAX_FETCHED_RECORDS per request. When a kind
+# exhausts its budget the response is flagged ``truncated`` and the returned
+# stream is clamped to a clean time cut so clients see a complete window with
+# older records omitted, not per-kind holes.
+_MAX_FETCHED_RECORDS = 5000
Review Comment:
Addressed by subsequent commits; the flagged code is no longer current after the rebase onto master. Resolving as outdated.
##########
superset/versioning/activity/queries.py:
##########
@@ -0,0 +1,754 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""DB-touching helpers for the activity-view read path.
+
+All Phase A relationship walks (``charts_attached_to_dashboard``,
+``datasets_used_by_chart``, ``batch_datasets_used_by_charts``),
+the Phase B change-record fetch (``fetch_change_records`` /
+``_select_change_rows_for_kinds``), the name-denormalization helpers
+(``_resolve_names_for_kind`` / ``apply_entity_name_denormalization``), the
+path-entity resolution helper (``resolve_path_entity``), and the
+tombstone-state lookup (``check_entity_tombstones``) live here.
+
+Each helper is a thin SELECT-and-shape function — no orchestration,
+no business logic. Callers in :mod:`scope`, :mod:`render`, and
+:mod:`orchestrator` compose them into the end-to-end request.
+
+**Inline imports.** Continuum's ``version_class`` / ``versioning_manager``
+and the Superset model classes are imported inside each helper because
+this package is loaded from ``init_versioning()`` before all SQLAlchemy
+mappers are configured.
+"""
+
+from __future__ import annotations
+
+import logging
+from datetime import datetime
+from typing import Any
+from uuid import UUID
+
+import sqlalchemy as sa
+from flask_appbuilder import Model
+
+from superset.extensions import db
+from superset.versioning.activity.kinds import (
+ API_KIND_TO_TABLE,
+ chunked_ids,
+ EntityWindows,
+ load_live_model,
+ NAME_COLUMN,
+ NOT_FOUND_EXC,
+ TABLE_KIND_TO_API,
+ Window,
+)
+from superset.versioning.activity.windows import row_within_any_window
+from superset.versioning.changes import version_changes_table
+
+logger = logging.getLogger(__name__)
+
+# ---- Path-entity resolution -----------------------------------------------
+
+
+def resolve_path_entity(model_cls: type[Model], entity_uuid: UUID) -> tuple[Any, int]:
+ """Resolve *entity_uuid* to ``(live_entity, entity_id)`` or raise a
+ typed 404.
+
+ Soft-delete handling is inherited transparently from
+ :func:`superset.versioning.queries.find_active_by_uuid` once it
+ learns to filter out ``deleted_at IS NOT NULL`` rows; at that point
+ soft-deleted paths will also raise here.
+ """
+ # pylint: disable=import-outside-toplevel
+ from superset.versioning.queries import find_active_by_uuid
+
+ entity = find_active_by_uuid(model_cls, entity_uuid)
+ if entity is None:
+ api_kind = model_cls.__name__
+ exc_cls = NOT_FOUND_EXC.get(api_kind)
+ if exc_cls is None:
+ raise LookupError(
+ f"Activity view does not support model class {api_kind!r}"
+ )
+ raise exc_cls(str(entity_uuid))
+ return entity, entity.id
+
+
+def first_tracked_tx(
+ model_cls: type[Model], entity_id: int, entity_uuid: UUID
+) -> int | None:
+ """Return the earliest Continuum transaction_id of the shadow rows
+ that belong to *this* live entity, matched on ``(id, uuid)``; ``None``
+ when the entity has no tracked history yet.
+
+ Used to lower-bound the self-scope window. Matching on the integer
+ ``id`` alone would inherit a previously hard-deleted entity's history
+ under id reuse (SQLite/MySQL reuse ``max(id)+1``); pinning the uuid
+ too — the same discrimination :func:`mark_first_tracked_saves` uses —
+ scopes the window to the current entity's own transactions.
+ """
+ # pylint: disable=import-outside-toplevel
+ from sqlalchemy_continuum import version_class
+
+ shadow_tbl = version_class(model_cls).__table__
+ stmt = sa.select(sa.func.min(shadow_tbl.c.transaction_id)).where(
+ shadow_tbl.c.id == entity_id,
+ shadow_tbl.c.uuid == entity_uuid,
+ )
+ return db.session.connection().execute(stmt).scalar()
+
+
+# ---- Phase A: relationship-traversal queries ------------------------------
+
+
+def charts_attached_to_dashboard(dashboard_id: int) -> list[tuple[int, Window]]:
+ """Return ``(slice_id, window)`` for every chart that has ever been on
+ *dashboard_id*, with each association's validity window in
+ transaction-id space.
+
+ Reads from ``dashboard_slices_version`` (Continuum's auto-generated
+ M2M shadow). Rows with ``operation_type = 2`` (DELETE) are excluded
+ so we don't synthesize a phantom window from a detachment row.
+ """
+ # pylint: disable=import-outside-toplevel
+ from sqlalchemy_continuum import version_class
+
+ from superset.models.dashboard import Dashboard
+
+ metadata = version_class(Dashboard).__table__.metadata
+ m2m_tbl = metadata.tables.get("dashboard_slices_version")
+ if m2m_tbl is None:
+ return []
+
+ rows = (
+ db.session.connection()
+ .execute(
+ sa.select(
+ m2m_tbl.c.slice_id,
+ m2m_tbl.c.transaction_id,
+ m2m_tbl.c.end_transaction_id,
+ ).where(
+ m2m_tbl.c.dashboard_id == dashboard_id,
+ m2m_tbl.c.operation_type != 2,
+ m2m_tbl.c.slice_id.is_not(None),
+ )
+ )
+ .all()
+ )
+ result: list[tuple[int, Window]] = []
+ for row in rows:
+ try:
+ window = Window(row[1], row[2])
+ except ValueError:
+ # A degenerate shadow row (end_tx <= start_tx) must not 500 the
+ # endpoint; skip it and leave a breadcrumb for investigation.
+ logger.warning(
+ "activity: skipping degenerate dashboard_slices_version row "
+ "(dashboard_id=%s, slice_id=%s, tx=%s, end_tx=%s)",
+ dashboard_id,
+ row[0],
+ row[1],
+ row[2],
+ )
+ continue
+ result.append((row[0], window))
+ return result
+
+
+def datasets_used_by_chart(slice_id: int) -> list[tuple[int, Window]]:
+ """Return ``(datasource_id, window)`` for every dataset that *slice_id*
+ has ever pointed at, with each association's validity window.
+
+ Single-slice form, used by ``_resolve_chart_scope`` where there
+ is only one chart to walk. The dashboard-scope path calls
+ :func:`batch_datasets_used_by_charts` instead so the query fires
+ once for all slices on the dashboard, not once per slice.
+
+ Reads from ``slices_version`` (the chart parent shadow). Filters to
+ ``datasource_type = 'table'`` because the activity view only follows
+ the chart → ``SqlaTable`` dependency edge (not legacy/other
+ datasources). Rows with ``operation_type = 2`` are excluded.
+ """
+ return batch_datasets_used_by_charts({slice_id}).get(slice_id, [])
+
+
+def batch_datasets_used_by_charts(
+ slice_ids: set[int],
+) -> dict[int, list[tuple[int, Window]]]:
+ """Batch form of :func:`datasets_used_by_chart`. Returns
+ ``{slice_id: [(dataset_id, window), ...]}`` in a single query so the
+ dashboard-scope walker doesn't fire one query per chart on the
+ dashboard. The previous per-slice shape became O(n_charts) round-
+ trips, which dominated ``get_activity`` latency on dashboards with
+ rich history (profile run 2026-05-26 showed `resolve_scope`
+ accounting for ~1.9s out of 4s p95).
+ """
+ if not slice_ids:
+ return {}
+
+ # pylint: disable=import-outside-toplevel
+ from sqlalchemy_continuum import version_class
+
+ from superset.models.slice import Slice
+
+ slices_tbl = version_class(Slice).__table__
+ grouped: dict[int, list[tuple[int, Window]]] = {}
+ # Chunk the IN-clause under SQLite's bind-variable floor (a dashboard can
+ # carry more charts than the floor allows in one statement).
+ rows: list[Any] = []
+ for chunk in chunked_ids(slice_ids):
+ rows.extend(
+ db.session.connection()
+ .execute(
+ sa.select(
+ slices_tbl.c.id,
+ slices_tbl.c.datasource_id,
+ slices_tbl.c.transaction_id,
+ slices_tbl.c.end_transaction_id,
+ ).where(
+ slices_tbl.c.id.in_(chunk),
+ slices_tbl.c.datasource_type == "table",
+ slices_tbl.c.operation_type != 2,
+ slices_tbl.c.datasource_id.is_not(None),
+ )
+ )
+ .mappings()
+ .all()
+ )
+ for row in rows:
+ try:
+ window = Window(row["transaction_id"], row["end_transaction_id"])
+ except ValueError:
+ # A degenerate shadow row (end_tx <= start_tx) must not 500 the
+ # endpoint; skip it and leave a breadcrumb for investigation.
+ logger.warning(
+ "activity: skipping degenerate slices_version row "
+ "(slice_id=%s, datasource_id=%s, tx=%s, end_tx=%s)",
+ row["id"],
+ row["datasource_id"],
+ row["transaction_id"],
+ row["end_transaction_id"],
+ )
+ continue
+ grouped.setdefault(row["id"], []).append((row["datasource_id"], window))
+ return grouped
+
+
+# ---- Phase B: change-record fetch -----------------------------------------
+
+
+def fetch_change_records(
+ entity_window_tuples: list[EntityWindows],
+ since: datetime | None,
+ until: datetime | None,
+) -> tuple[list[dict[str, Any]], bool]:
+ """Fetch all ``version_changes`` rows matching any of the supplied
+ entity-window tuples, joined with ``version_transaction`` for
+ ``issued_at`` and ``user_id``.
+
+ Each tuple is ``(api_kind, entity_id, [(start_tx, end_tx), ...])``;
+ a record matches when ``entity_kind`` equals the table-stored form
+ of *api_kind*, ``entity_id`` matches, and ``transaction_id`` falls
+ inside at least one of the entity's windows. ``since``/``until``
+ further restrict by ``issued_at``.
+
+ Implementation: one SELECT per kind with ``entity_id IN (...)`` and
+ a wide ``transaction_id`` bound (the union of all windows for that
+ kind). Per-window precision is applied in Python afterward. This
+ keeps the SQL shape proportional to the number of *kinds* (≤3) and
+ the bound proportional to the union of windows, not the cross-
+ product of (entity, window) — which previously generated one OR
+ clause per (entity, window) pair and hit SQLite's
+ ``SQLITE_MAX_EXPR_DEPTH`` limit on dashboards with many slices
+ or many historical attachment windows.
+
+ The visibility filter runs after this function (records
+ the requester can't read are silently dropped and must not
+ contribute to ``count``), so the orchestrator paginates in Python
+ over the filtered list — there is no DB-level page ``OFFSET`` here.
+ There IS a per-kind safety ceiling (``_MAX_FETCHED_RECORDS``) that
+ bounds how much history a single request materializes (to at most
+ ``n_kinds * _MAX_FETCHED_RECORDS``); when a kind exhausts it, the
+ second return value is ``True``, the returned stream is clamped to a
+ clean time cut (records older than the highest per-kind fetch floor
+ are dropped so truncation is a time boundary, not per-kind holes), and
+ the caller surfaces ``truncated`` on the response.
+
+ Returns ``(records, truncated)``. Records are ordered by
+ ``(issued_at DESC, transaction_id DESC, sequence DESC)`` — the
+ secondary keys break ties for the stable-ordering contract.
+ """
+ if not entity_window_tuples:
+ return [], False
+
+ # Group windows by (table_kind, entity_id) and by table_kind for SQL
+ # narrowing. The fetch is per-kind; the post-filter is per-entity.
+ windows_by_entity: dict[tuple[str, int], list[Window]] = {}
+ ids_by_kind: dict[str, set[int]] = {}
+ for api_kind, entity_id, windows in entity_window_tuples:
+ table_kind = API_KIND_TO_TABLE.get(api_kind)
+ if table_kind is None or not windows:
+ continue
+ ids_by_kind.setdefault(table_kind, set()).add(entity_id)
+ windows_by_entity.setdefault((table_kind, entity_id), []).extend(windows)
+
+ if not ids_by_kind:
+ return [], False
+
+ # Per-kind transaction_id bounds = the union of that kind's windows.
+ # Pushing these into the SQL WHERE ensures the per-statement LIMIT
+ # selects from IN-WINDOW rows. Without it, a related entity whose
+ # association window is far in the past would have the newest ``limit``
+ # (out-of-window) rows fetched and discarded, silently dropping its
+ # in-window records that lie beyond the limit. ``end_tx = None``
+ # (open-ended/current) means no upper bound for that kind.
+ bounds_by_kind: dict[str, tuple[int, int | None]] = {}
+ for (table_kind, _entity_id), windows in windows_by_entity.items():
+ for w in windows:
+ cur = bounds_by_kind.get(table_kind)
+ if cur is None:
+ bounds_by_kind[table_kind] = (w.start_tx, w.end_tx)
+ continue
+ lo = min(cur[0], w.start_tx)
+ hi = None if (cur[1] is None or w.end_tx is None) else max(cur[1], w.end_tx)
+ bounds_by_kind[table_kind] = (lo, hi)
+
+ rows, truncated, truncation_floor = _select_change_rows_for_kinds(
+ ids_by_kind, bounds_by_kind, since, until, _MAX_FETCHED_RECORDS
+ )
+ filtered = [
+ row
+ for row in rows
+ if row_within_any_window(
+ row, windows_by_entity.get((row["entity_kind"], row["entity_id"]), [])
+ )
+ ]
+ if truncated and truncation_floor is not None:
+ # Collapse truncation to a clean time cut. Below the highest per-kind
+ # fetch floor at least one kind is missing rows, so surfacing the
+ # other kinds there would present an incomplete ("nothing else
+ # changed") picture. Drop everything older than the floor so
+ # ``truncated=True`` means "complete at/after this instant, nothing
+ # shown before it" — a time boundary rather than per-kind holes.
+ filtered = [r for r in filtered if r["issued_at"] >= truncation_floor]
+ # Sort key must be TOTAL so pagination is stable across requests: two
+ # records from different entities can share (issued_at, transaction_id,
+ # sequence), so append (entity_kind, entity_id) to break remaining ties
+ # deterministically. Without these the relative order of tied records
+ # depends on set-iteration order and a record could shift pages.
+ filtered.sort(
+ key=lambda r: (
+ r["issued_at"],
+ r["transaction_id"],
+ r["sequence"],
+ r["entity_kind"],
+ r["entity_id"],
+ ),
+ reverse=True,
+ )
+ return filtered, truncated
+
+
+def _select_change_rows_for_kinds(
+ ids_by_kind: dict[str, set[int]],
+ bounds_by_kind: dict[str, tuple[int, int | None]],
+ since: datetime | None,
+ until: datetime | None,
+ limit: int,
+) -> tuple[list[dict[str, Any]], bool, datetime | None]:
+ """Fire one SELECT per entity_kind with ``entity_id IN (...)``;
+ concatenate the results. Each SELECT joins ``version_transaction``
+ + ``ab_user`` so the orchestrator has the columns it needs for
+ decoration.
+
+ Per-kind, not one query: SQLAlchemy's ``tuple_(entity_kind,
+ entity_id).in_(...)`` would collapse the three queries into one,
+ but its SQL emission is not portable across Postgres, MySQL, and
+ SQLite. The per-kind shape is the correct trade-off given
+ Superset's multi-dialect requirement (at most 3 round-trips per
+ request, bounded by the kind taxonomy). Do not "optimise" into a
+ composite-tuple IN clause without verifying the SQL on all three
+ dialects.
+
+ **Init-order dependency.** ``tx_tbl.c.action_kind`` resolves only
+ after ``init_versioning()`` has run — the column is appended onto
+ Continuum's transaction Table by
+ ``superset.versioning.factory.VersionTransactionFactory`` at app
+ start via ``append_column`` + ``add_property``. This helper is
+ safe to call from request-path code because the app is fully
+ initialised by then; calling it from a script that imports the
+ versioning package without going through ``init_versioning()``
+ will raise ``AttributeError`` on the ``action_kind`` attribute
+ access below."""
+ # pylint: disable=import-outside-toplevel
+ from sqlalchemy_continuum import versioning_manager
+
+ from superset import security_manager
+
+ tx_tbl = versioning_manager.transaction_cls.__table__
+ user_tbl = security_manager.user_model.__table__
+ vc = version_changes_table
+ join_tree = vc.join(tx_tbl, vc.c.transaction_id == tx_tbl.c.id).outerjoin(
+ user_tbl, tx_tbl.c.user_id == user_tbl.c.id
+ )
+ select_cols = (
+ vc.c.transaction_id,
+ vc.c.entity_kind,
+ vc.c.entity_id,
+ vc.c.sequence,
+ vc.c.kind,
+ vc.c.operation,
+ vc.c.path,
+ vc.c.from_value,
+ vc.c.to_value,
+ tx_tbl.c.issued_at,
+ tx_tbl.c.user_id,
+ # ``action_kind`` is the high-level avenue (restore / import /
+ # clone / NULL=ordinary save) stamped by the originating
+ # command via the change-record listener. All records sharing a
+ # ``transaction_id`` share the same value. The column is
+ # declared on the Continuum Table by ``VersionTransactionFactory``,
+ # so ``tx_tbl.c.action_kind`` resolves cleanly here. See
+ # the three change-record dimensions.
+ tx_tbl.c.action_kind,
+ user_tbl.c.id.label("changed_by_id"),
+ user_tbl.c.first_name,
+ user_tbl.c.last_name,
+ )
+
+ out: list[dict[str, Any]] = []
+ truncated = False
+ truncation_floors: list[datetime] = []
+ for table_kind, entity_ids in ids_by_kind.items():
+ # The fetch budget is per KIND, not per id-chunk. A kind whose
+ # entity_ids span several bind-variable chunks shares one ``limit``
+ # across those chunks, so the rows a single request materializes are
+ # bounded by ``n_kinds * limit`` (<= 3 * _MAX_FETCHED_RECORDS), not
+ # ``n_chunks * limit``. ``entity_ids`` is chunked to stay inside
+ # SQLite's ``SQLITE_MAX_VARIABLE_NUMBER`` floor (default 999 in many
+ # builds); Postgres + MySQL accept the full list but the chunk is
+ # dialect-agnostic for simplicity.
+ kind_rows: list[dict[str, Any]] = []
+ for chunk in chunked_ids(entity_ids):
+ remaining = limit - len(kind_rows)
+ if remaining <= 0:
+ break
+ stmt = (
+ sa.select(*select_cols)
+ .select_from(join_tree)
+ .where(
+ vc.c.entity_kind == table_kind,
+ vc.c.entity_id.in_(chunk),
+ )
+ )
+ # Bound by the kind's window union so the LIMIT picks in-window
+ # rows (see fetch_change_records). The per-entity window filter
+ # still runs in Python afterwards for exact membership.
+ tx_lo, tx_hi = bounds_by_kind[table_kind]
+ stmt = stmt.where(vc.c.transaction_id >= tx_lo)
+ if tx_hi is not None:
+ stmt = stmt.where(vc.c.transaction_id < tx_hi)
+ if since is not None:
+ stmt = stmt.where(tx_tbl.c.issued_at >= since)
+ if until is not None:
+ stmt = stmt.where(tx_tbl.c.issued_at < until)
+ # Match fetch_change_records' final Python sort key order
+ # (issued_at, transaction_id, sequence, entity_id) so the
+ # per-kind budget keeps exactly the rows the final sort ranks
+ # highest. entity_kind is constant within a per-kind statement.
+ stmt = stmt.order_by(
+ tx_tbl.c.issued_at.desc(),
+ vc.c.transaction_id.desc(),
+ vc.c.sequence.desc(),
+ vc.c.entity_id.desc(),
+ ).limit(remaining)
+ kind_rows.extend(
+ dict(row)
+ for row in db.session.connection().execute(stmt).mappings().all()
+ )
+ if len(kind_rows) >= limit:
+ # This kind exhausted its budget; older rows almost certainly
+ # exist beyond it. Record the oldest ``issued_at`` fetched for
+ # the kind so the caller can collapse truncation into a single
+ # clean time cut across all kinds (see fetch_change_records).
+ truncated = True
+ truncation_floors.append(min(r["issued_at"] for r in kind_rows))
+ out.extend(kind_rows)
+ truncation_floor = max(truncation_floors) if truncation_floors else None
+ return out, truncated, truncation_floor
+
+
+# Per-KIND safety ceiling on how many change rows a single activity request
+# will materialize for one entity kind (shared across that kind's id-chunks).
+# Bounds memory/CPU for a path entity with very long history or many related
+# entities to <= n_kinds * _MAX_FETCHED_RECORDS per request. When a kind
+# exhausts its budget the response is flagged ``truncated`` and the returned
+# stream is clamped to a clean time cut so clients see a complete window with
+# older records omitted, not per-kind holes.
+_MAX_FETCHED_RECORDS = 5000
+
+
+def mark_first_tracked_saves(records: list[dict[str, Any]]) -> None:
+ """Set ``first_tracked_save`` on each record in place: ``True`` when
+ the record's transaction is the entity's FIRST UPDATE (op=1) in its
+ shadow table.
+
+ The first save of an entity that predates versioning replays every
+ params-normalization delta against the retroactive baseline — a
+ legacy chart's first Explore save produced ~74 records in one
+ transaction (version-history UI feedback). The server
+ can't distinguish "normalization" from "the user changed 74 things",
+ but it CAN say "this was the entity's first tracked save"; clients
+ use the marker to collapse such transactions.
+
+ One ``GROUP BY`` query per kind (≤3), chunked like the record fetch.
+ Shadow rows are matched on ``(id, uuid)`` against the live row — a
+ bare ``id`` match would inherit a previously hard-deleted entity's
+ history under id reuse (SQLite/MySQL reuse ``max(id)+1``) and mark
+ the wrong transaction. Consequence: hard-deleted entities (no live
+ row) and NULL-uuid shadow rows never get a ``True`` marker — their
+ records always carry ``first_tracked_save=False``. Mutates *records*
+ in place — same contract as the other decoration passes in
+ :mod:`superset.versioning.activity.render`.
+ """
+ if not records:
+ return
+ # pylint: disable=import-outside-toplevel
+ from sqlalchemy_continuum import version_class
+
+ ids_by_kind: dict[str, set[int]] = {}
+ for r in records:
+ ids_by_kind.setdefault(r["entity_kind"], set()).add(r["entity_id"])
+
+ first_tx_by_entity: dict[tuple[str, int], int] = {}
+ for table_kind, entity_ids in ids_by_kind.items():
+ model_name = TABLE_KIND_TO_API.get(table_kind)
+ if model_name is None:
+ continue
+ live_model = load_live_model(model_name)
+ live_tbl = live_model.__table__
+ shadow_tbl = version_class(live_model).__table__
+ for chunk in chunked_ids(entity_ids):
+ stmt = (
+ sa.select(
+ shadow_tbl.c.id,
+ sa.func.min(shadow_tbl.c.transaction_id),
+ )
+ .select_from(
+ shadow_tbl.join(
+ live_tbl,
+ sa.and_(
+ shadow_tbl.c.id == live_tbl.c.id,
+ shadow_tbl.c.uuid == live_tbl.c.uuid,
+ ),
+ )
+ )
+ .where(
+ shadow_tbl.c.operation_type == 1,
+ shadow_tbl.c.id.in_(chunk),
+ )
+ .group_by(shadow_tbl.c.id)
+ )
+ for entity_id, min_tx in db.session.connection().execute(stmt):
+ first_tx_by_entity[(table_kind, entity_id)] = min_tx
+
+ for r in records:
+ r["first_tracked_save"] = (
+ first_tx_by_entity.get((r["entity_kind"], r["entity_id"]))
+ == r["transaction_id"]
+ )
+
+
+# ---- Name denormalization -------------------------------------------------
+
+
+def _resolve_names_for_kind(
+ api_kind: str, pairs: set[tuple[int, int]]
+) -> dict[tuple[int, int], str]:
+ """For one entity kind, return ``{(entity_id, target_tx): name}`` from
+ the shadow row valid at *target_tx* (validity-strategy predicate).
+ Empty mapping when the kind has no name column registered.
+ """
+ # pylint: disable=import-outside-toplevel
+ from sqlalchemy_continuum import version_class
+
+ if api_kind not in NAME_COLUMN or not pairs:
+ return {}
+
+ model_name, name_col = NAME_COLUMN[api_kind]
+ model_cls = load_live_model(model_name)
+ shadow_tbl = version_class(model_cls).__table__
+ ids = {eid for eid, _ in pairs}
+ # Bound the scan to the transaction range the page-set actually needs.
+ # A row valid at some target_tx must start at/before the newest target
+ # and stay open past the oldest one, so rows outside
+ # ``[min_target_tx, max_target_tx]`` can never win the validity match.
+ # Without this bound the query loads every historical version row of
+ # every referenced entity — an entity with thousands of versions would
+ # defeat the memory ceiling the change-record fetch enforces.
+ target_txs = {target_tx for _, target_tx in pairs}
+ min_tx, max_tx = min(target_txs), max(target_txs)
+ per_entity: dict[int, list[tuple[int, int | None, Any]]] = {}
+ # Chunk the IN-clause to stay under SQLite's bind-variable floor (the
+ # same reason _select_change_rows_for_kinds chunks).
+ for chunk in chunked_ids(ids):
+ rows = (
+ db.session.connection()
+ .execute(
+ sa.select(
+ shadow_tbl.c.id,
+ shadow_tbl.c.transaction_id,
+ shadow_tbl.c.end_transaction_id,
+ shadow_tbl.c[name_col],
+ ).where(
+ shadow_tbl.c.id.in_(chunk),
+ shadow_tbl.c.transaction_id <= max_tx,
+ sa.or_(
+ shadow_tbl.c.end_transaction_id.is_(None),
+ shadow_tbl.c.end_transaction_id > min_tx,
+ ),
+ )
+ )
+ .all()
+ )
+ for row in rows:
+ per_entity.setdefault(row[0], []).append((row[1], row[2], row[3]))
+
+ resolved: dict[tuple[int, int], str] = {}
+ for entity_id, target_tx in pairs:
+ for start_tx, end_tx, name in per_entity.get(entity_id, []):
+ if start_tx <= target_tx and (end_tx is None or end_tx > target_tx):
+ # Coerce a NULL name (e.g. a DELETE shadow row winning the
+ # validity match) to "" so entity_name is never None.
+ resolved[(entity_id, target_tx)] = name if name is not None else ""
+ break
+ return resolved
+
+
+def apply_entity_name_denormalization(records: list[dict[str, Any]]) -> None:
+ """Resolve each record's ``entity_name`` from the shadow row valid at
+ its ``transaction_id``. Adds an ``entity_name`` key to every record
+ in place; returns ``None``.
+
+ The lookup is per (table-stored ``entity_kind``, ``entity_id``,
+ ``transaction_id``) triple. One ``IN``-clause query per kind keeps
+ round-trips bounded by the number of distinct kinds (≤3) regardless
+ of result-set size. The in-place mutation avoids re-allocating
+ thousands of dicts on hot dashboards; the name + return signature
+ make the side effect explicit instead of pretending to be a pure
+ projection.
+ """
+ if not records:
+ return
+
+ needed_by_kind: dict[str, set[tuple[int, int]]] = {}
+ for record in records:
+ api_kind = TABLE_KIND_TO_API.get(record["entity_kind"])
+ if api_kind is None or api_kind not in NAME_COLUMN:
+ continue
+ needed_by_kind.setdefault(api_kind, set()).add(
+ (record["entity_id"], record["transaction_id"])
+ )
+
+ resolved: dict[tuple[str, int, int], str] = {}
+ for api_kind, pairs in needed_by_kind.items():
+ for (entity_id, target_tx), name in _resolve_names_for_kind(
+ api_kind, pairs
+ ).items():
+ resolved[(api_kind, entity_id, target_tx)] = name
+
+ for record in records:
+ api_kind_for_record = TABLE_KIND_TO_API.get(record["entity_kind"], "")
+ key = (api_kind_for_record, record["entity_id"], record["transaction_id"])
+ record["entity_name"] = resolved.get(key, "")
+
+
+# ---- Live-row existence + soft-delete state -------------------------------
+
+
+def check_entity_tombstones(
+ distinct_entities: set[tuple[str, int]],
+) -> dict[tuple[str, int], dict[str, Any]]:
+ """For each ``(api_kind, entity_id)``, report ``deleted`` (no live
+ row) and ``deletion_state`` (``"soft_deleted"`` iff the live row has
+ a non-null ``deleted_at``, else ``None``).
+
+ The ``deleted_at`` column is probed for at runtime: when the model
+ classes don't have one, entities are reported as never soft-deleted
+ (``deletion_state=None``); when a ``deleted_at`` column exists, this
+ helper picks it up automatically.
+ """
+ result: dict[tuple[str, int], dict[str, Any]] = {}
+ if not distinct_entities:
+ return result
+
+ by_kind: dict[str, list[int]] = {}
+ for api_kind, entity_id in distinct_entities:
+ by_kind.setdefault(api_kind, []).append(entity_id)
+
+ # ``no_autoflush`` mirrors the defensive posture of the listener-
+ # side reads. Today's callers run from request-path code with no
+ # pending writes; a future caller that probes tombstones before a
+ # flush would otherwise trigger autoflush mid-read.
+ with db.session.no_autoflush:
+ for api_kind, entity_ids in by_kind.items():
+ for entity_id, state in _tombstone_states_for_kind(
+ api_kind, entity_ids
+ ).items():
+ result[(api_kind, entity_id)] = state
+ return result
+
+
+_TOMBSTONE = {"deleted": True, "deletion_state": None}
Review Comment:
Addressed by subsequent commits; the flagged code is no longer current after the rebase onto master. Resolving as outdated.
##########
superset/versioning/activity/scope.py:
##########
@@ -0,0 +1,118 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Scope resolution — turn a path entity into the related-entity walk.
+
+Composes :mod:`~superset.versioning.activity.queries` (Phase A
+relationship walks) and :mod:`~superset.versioning.activity.windows`
+(pure interval arithmetic) into the
+``list[EntityWindows]`` scope that
+:func:`~superset.versioning.activity.queries.fetch_change_records`
+consumes.
+
+The functions here read the DB (via the Phase A helpers in
+:mod:`~superset.versioning.activity.queries`); the pure window-
+arithmetic functions previously colocated here now live in
+:mod:`~superset.versioning.activity.windows` so the package no longer
+needs a lazy import to dodge a ``scope ↔ queries`` cycle.
+"""
+
+from __future__ import annotations
+
+from superset.versioning.activity.kinds import EntityWindows, Window
+from superset.versioning.activity.queries import (
+ batch_datasets_used_by_charts,
+ charts_attached_to_dashboard,
+ datasets_used_by_chart,
+)
+from superset.versioning.activity.windows import (
+ intersect_windows,
+ merge_entity_windows,
+)
+
+
+def resolve_scope(
+ path_kind: str,
+ path_id: int,
+ include: str,
+ self_start_tx: int | None = None,
+) -> list[EntityWindows]:
+ """Build the ``[(api_kind, entity_id, [windows])]`` list that
+ :func:`~superset.versioning.activity.queries.fetch_change_records`
+ consumes, branching by *path_kind* and *include* mode.
+
+ *self_start_tx* lower-bounds the self window at the path entity's first
+ tracked transaction (see
+ :func:`~superset.versioning.activity.queries.first_tracked_tx`). This
+ scopes the self stream to the current entity's own history: matching
+ self records on the bare integer id would otherwise inherit a
+ hard-deleted predecessor's change records under id reuse
+ (SQLite/MySQL reuse ``max(id)+1``). ``None`` means the entity has no
+ tracked history yet, so the self stream contributes nothing.
+ """
+ want_self = include in ("all", "self")
Review Comment:
Addressed by subsequent commits; the flagged code is no longer current after the rebase onto master. Resolving as outdated.
##########
superset/versioning/activity/scope.py:
##########
@@ -0,0 +1,118 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Scope resolution — turn a path entity into the related-entity walk.
+
+Composes :mod:`~superset.versioning.activity.queries` (Phase A
+relationship walks) and :mod:`~superset.versioning.activity.windows`
+(pure interval arithmetic) into the
+``list[EntityWindows]`` scope that
+:func:`~superset.versioning.activity.queries.fetch_change_records`
+consumes.
+
+The functions here read the DB (via the Phase A helpers in
+:mod:`~superset.versioning.activity.queries`); the pure window-
+arithmetic functions previously colocated here now live in
+:mod:`~superset.versioning.activity.windows` so the package no longer
+needs a lazy import to dodge a ``scope ↔ queries`` cycle.
+"""
+
+from __future__ import annotations
+
+from superset.versioning.activity.kinds import EntityWindows, Window
+from superset.versioning.activity.queries import (
+ batch_datasets_used_by_charts,
+ charts_attached_to_dashboard,
+ datasets_used_by_chart,
+)
+from superset.versioning.activity.windows import (
+ intersect_windows,
+ merge_entity_windows,
+)
+
+
+def resolve_scope(
+ path_kind: str,
+ path_id: int,
+ include: str,
+ self_start_tx: int | None = None,
+) -> list[EntityWindows]:
+ """Build the ``[(api_kind, entity_id, [windows])]`` list that
+ :func:`~superset.versioning.activity.queries.fetch_change_records`
+ consumes, branching by *path_kind* and *include* mode.
+
+ *self_start_tx* lower-bounds the self window at the path entity's first
+ tracked transaction (see
+ :func:`~superset.versioning.activity.queries.first_tracked_tx`). This
+ scopes the self stream to the current entity's own history: matching
+ self records on the bare integer id would otherwise inherit a
+ hard-deleted predecessor's change records under id reuse
+ (SQLite/MySQL reuse ``max(id)+1``). ``None`` means the entity has no
+ tracked history yet, so the self stream contributes nothing.
+ """
+ want_self = include in ("all", "self")
+ want_related = include in ("all", "related")
Review Comment:
Addressed by subsequent commits; the flagged code is no longer current after the rebase onto master. Resolving as outdated.
##########
superset/versioning/activity/scope.py:
##########
@@ -0,0 +1,118 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Scope resolution — turn a path entity into the related-entity walk.
+
+Composes :mod:`~superset.versioning.activity.queries` (Phase A
+relationship walks) and :mod:`~superset.versioning.activity.windows`
+(pure interval arithmetic) into the
+``list[EntityWindows]`` scope that
+:func:`~superset.versioning.activity.queries.fetch_change_records`
+consumes.
+
+The functions here read the DB (via the Phase A helpers in
+:mod:`~superset.versioning.activity.queries`); the pure window-
+arithmetic functions previously colocated here now live in
+:mod:`~superset.versioning.activity.windows` so the package no longer
+needs a lazy import to dodge a ``scope ↔ queries`` cycle.
+"""
+
+from __future__ import annotations
+
+from superset.versioning.activity.kinds import EntityWindows, Window
+from superset.versioning.activity.queries import (
+ batch_datasets_used_by_charts,
+ charts_attached_to_dashboard,
+ datasets_used_by_chart,
+)
+from superset.versioning.activity.windows import (
+ intersect_windows,
+ merge_entity_windows,
+)
+
+
+def resolve_scope(
+ path_kind: str,
+ path_id: int,
+ include: str,
+ self_start_tx: int | None = None,
+) -> list[EntityWindows]:
+ """Build the ``[(api_kind, entity_id, [windows])]`` list that
+ :func:`~superset.versioning.activity.queries.fetch_change_records`
+ consumes, branching by *path_kind* and *include* mode.
+
+ *self_start_tx* lower-bounds the self window at the path entity's first
+ tracked transaction (see
+ :func:`~superset.versioning.activity.queries.first_tracked_tx`). This
+ scopes the self stream to the current entity's own history: matching
+ self records on the bare integer id would otherwise inherit a
+ hard-deleted predecessor's change records under id reuse
+ (SQLite/MySQL reuse ``max(id)+1``). ``None`` means the entity has no
+ tracked history yet, so the self stream contributes nothing.
+ """
+ want_self = include in ("all", "self")
+ want_related = include in ("all", "related")
+
+ scope: list[EntityWindows] = []
+ if want_self and self_start_tx is not None:
+ scope.append((path_kind, path_id, [Window(self_start_tx, None)]))
+ if want_related:
+ scope.extend(_resolve_related_scope(path_kind, path_id))
+ return scope
+
+
+def _resolve_related_scope(path_kind: str, path_id: int) -> list[EntityWindows]:
+ """Walk the dependency edges from the path entity to its related
+ entities. Datasets have no transitive layer in V2."""
+ if path_kind == "Dashboard":
+ return _resolve_dashboard_scope(path_id)
+ if path_kind == "Slice":
+ return _resolve_chart_scope(path_id)
+ return []
+
+
+def _resolve_dashboard_scope(dashboard_id: int) -> list[EntityWindows]:
+ """Charts on the dashboard during their attachment window, plus
+ datasets each chart pointed at during the intersection of (chart-
+ attachment, chart-on-dataset)."""
+ scope: list[EntityWindows] = []
+ chart_windows: dict[int, list[Window]] = {}
+ for slice_id, window in charts_attached_to_dashboard(dashboard_id):
+ chart_windows.setdefault(slice_id, []).append(window)
+
+ # One query for the dataset-history of every chart on the dashboard,
+ # not one query per chart. The per-slice form was O(n_charts) round-
+ # trips which dominated p95 on rich dashboards.
+ dataset_windows_by_slice = batch_datasets_used_by_charts(set(chart_windows))
Review Comment:
Fixed in 8b19744b85 by adding the requested explicit type annotation.
##########
superset/versioning/activity/__init__.py:
##########
@@ -0,0 +1,102 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Read-side queries for the cross-entity activity-view API.
+
+Companion to :mod:`superset.versioning.queries`. Whereas ``queries.py``
+returns transaction-level history for a single entity, the helpers
+here unify change-record history across an entity's transitive
+dependency chain — a dashboard's activity stream includes edits to
+charts that were attached to it AND edits to datasets those charts
+pointed at, each time-bounded by when the relationship was active.
+
+One public entry point — :func:`get_activity` — dispatches on the
+first argument to serve all three endpoint families:
+
+* ``get_activity(Dashboard, dashboard_uuid, ...)`` — own edits +
+ charts attached during their dashboard window + datasets those
+ charts used during their chart window.
+* ``get_activity(Slice, chart_uuid, ...)`` — own edits + datasets the
+ chart pointed at during association.
+* ``get_activity(SqlaTable, dataset_uuid, ...)`` — own edits only.
+ Datasets are not transitive recipients of activity in V2.
+
+Package layout (descends from public entry point to leaf helpers):
+
+* :mod:`.orchestrator` — :func:`get_activity` (public), the
+ ``activity_endpoint`` REST helper, the request param parser
+ (:func:`parse_activity_query_params`), and the observability
+ instrumentation (request-shape + per-kind metrics).
+* :mod:`.scope` — scope resolution (DB-touching):
+ :func:`resolve_scope` / :func:`_resolve_dashboard_scope` /
+ :func:`_resolve_chart_scope` / :func:`_resolve_related_scope`.
+* :mod:`.windows` — pure window arithmetic on half-open
+ ``[start_tx, end_tx)`` intervals: :func:`intersect_windows` /
+ :func:`union_windows` / :func:`merge_entity_windows` /
+ :func:`row_within_any_window`. Extracted from :mod:`.scope` so
+ :mod:`.queries` can import the pure helpers at module-top instead
+ of through a cycle-dodging lazy import.
+* :mod:`.queries` — every DB-touching helper: Phase A relationship
+ walks, Phase B change-record fetch, name denormalization,
+ path-entity resolution, and tombstone-state lookup.
+* :mod:`.impact` — per-record impact-count computation (the only
+ field that requires its own batched query).
+* :mod:`.visibility` — the silent visibility filter; uses
+ the same SQL-side access filters FAB applies on list endpoints.
+* :mod:`.render` — record-decoration helpers that turn raw rows into
+ the ActivityRecord DTO (summary headlines, ``changed_by`` projection,
+ uuid lookup).
+* :mod:`.kinds` — the kind-translation tables, the ``Window`` /
+ ``EntityWindows`` type aliases, and :func:`load_live_model`.
+
+The public surface (re-exported here) is the eight symbols below.
+Sub-module privates are intentionally NOT re-exported — tests and
+new internal callers should import them from their owning submodule
+(e.g. ``from superset.versioning.activity.windows import
+intersect_windows``) so the package's public API stays scannable.
+
+``PathEntityResponseError`` and ``resolve_endpoint_path_entity`` are
+re-exported here from :mod:`superset.versioning.api_helpers` (where
+they live alongside the ``/versions/`` endpoint handlers) so the
+three ``/activity/`` endpoint callers can ``from
+superset.versioning.activity import resolve_endpoint_path_entity``
+without crossing into the ``/versions/`` module name.
+"""
+
+from __future__ import annotations
+
+from superset.versioning.activity.kinds import EntityWindows, Window
+from superset.versioning.activity.orchestrator import (
+ activity_endpoint,
+ ActivityParamsError,
+ get_activity,
+ parse_activity_query_params,
+)
+from superset.versioning.api_helpers import (
+ PathEntityResponseError,
+ resolve_endpoint_path_entity,
+)
+
+__all__ = [
+ "ActivityParamsError",
+ "EntityWindows",
+ "PathEntityResponseError",
+ "Window",
+ "activity_endpoint",
+ "get_activity",
+ "parse_activity_query_params",
+ "resolve_endpoint_path_entity",
+]
Review Comment:
Addressed by subsequent commits; the flagged code is no longer current after the rebase onto master. Resolving as outdated.
##########
superset/versioning/activity/impact.py:
##########
@@ -0,0 +1,186 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Per-record impact computation for the activity DTO.
+
+Only dashboard-path activity records pointing at a ``SqlaTable``
+related entity carry an ``impact`` field — the number of charts on
+the dashboard at that transaction that were pointing at the dataset.
+This module computes that count in a single batched query per
+request:
+
+* :func:`collect_impact_pairs` — pulls the distinct
+ ``(dataset_id, transaction_id)`` pairs that need counts.
+* :func:`batch_chart_counts` — one SQL query joining
+ ``dashboard_slices_version`` and ``slices_version`` to count
+ the matching charts validity-strategy-style.
+* :func:`impact_for_record` — pure projection from the pre-fetched
+ counts onto each record (returns ``None`` for non-Dashboard paths
+ or non-SqlaTable kinds, matching the ``impact`` computation).
+
+Splitting the count batching from the pure projection keeps the SQL
+inside one function (the batched read) and the per-record decoration
+inside another (no DB).
+"""
+
+from __future__ import annotations
+
+from typing import Any
+
+import sqlalchemy as sa
+
+from superset.extensions import db
+from superset.versioning.activity.kinds import (
+ chunked_ids,
+ ENTITY_ID_CHUNK_SIZE,
+ TABLE_KIND_TO_API,
+)
+
+
+def collect_impact_pairs(
+ records: list[dict[str, Any]], path_kind: str
+) -> set[tuple[int, int]]:
+ """Distinct ``(dataset_id, transaction_id)`` pairs from *records*
+ that require an impact computation.
+
+ Only dashboard-path records whose related entity is a ``SqlaTable``
+ produce a non-null ``impact`` field; for any other shape this set
+ is empty and no DB query needs to fire.
+ """
+ if path_kind != "Dashboard":
+ return set()
+ return {
+ (record["entity_id"], record["transaction_id"])
+ for record in records
+ if TABLE_KIND_TO_API.get(record["entity_kind"]) == "SqlaTable"
+ }
+
+
+def batch_chart_counts(
+ dashboard_id: int, pairs: set[tuple[int, int]]
+) -> dict[tuple[int, int], int]:
+ """For every ``(dataset_id, target_tx)`` in *pairs*, count the
+ distinct charts that were both on *dashboard_id* and pointing at
+ *dataset_id* at *target_tx*.
+
+ One SELECT against ``dashboard_slices_version`` ⨝ ``slices_version``,
+ pulling the (slice, dataset, validity-window) state for every slice
+ ever on the dashboard whose dataset matches one of the requested
+ dataset_ids. The Python loop then applies the validity-strategy
+ predicate per pair. Replaces the previous N+1 shape that fired one
+ COUNT per related record.
+
+ Returns ``{(dataset_id, target_tx): count}``; pairs whose count
+ would be zero are omitted so the caller's ``.get(key, 0)`` is
+ correct.
+ """
+ if not pairs:
+ return {}
+
+ # pylint: disable=import-outside-toplevel
+ from sqlalchemy_continuum import version_class
+
+ from superset.models.slice import Slice
+
+ metadata = version_class(Slice).__table__.metadata
+ m2m_tbl = metadata.tables.get("dashboard_slices_version")
+ slices_tbl = version_class(Slice).__table__
+ if m2m_tbl is None:
+ return {}
+
+ dataset_ids = {dataset_id for dataset_id, _ in pairs}
Review Comment:
Addressed by subsequent commits; the flagged code is no longer current after the rebase onto master. Resolving as outdated.
##########
superset/versioning/activity/impact.py:
##########
@@ -0,0 +1,186 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Per-record impact computation for the activity DTO.
+
+Only dashboard-path activity records pointing at a ``SqlaTable``
+related entity carry an ``impact`` field — the number of charts on
+the dashboard at that transaction that were pointing at the dataset.
+This module computes that count in a single batched query per
+request:
+
+* :func:`collect_impact_pairs` — pulls the distinct
+ ``(dataset_id, transaction_id)`` pairs that need counts.
+* :func:`batch_chart_counts` — one SQL query joining
+ ``dashboard_slices_version`` and ``slices_version`` to count
+ the matching charts validity-strategy-style.
+* :func:`impact_for_record` — pure projection from the pre-fetched
+ counts onto each record (returns ``None`` for non-Dashboard paths
+ or non-SqlaTable kinds, matching the ``impact`` computation).
+
+Splitting the count batching from the pure projection keeps the SQL
+inside one function (the batched read) and the per-record decoration
+inside another (no DB).
+"""
+
+from __future__ import annotations
+
+from typing import Any
+
+import sqlalchemy as sa
+
+from superset.extensions import db
+from superset.versioning.activity.kinds import (
+ chunked_ids,
+ ENTITY_ID_CHUNK_SIZE,
+ TABLE_KIND_TO_API,
+)
+
+
+def collect_impact_pairs(
+ records: list[dict[str, Any]], path_kind: str
+) -> set[tuple[int, int]]:
+ """Distinct ``(dataset_id, transaction_id)`` pairs from *records*
+ that require an impact computation.
+
+ Only dashboard-path records whose related entity is a ``SqlaTable``
+ produce a non-null ``impact`` field; for any other shape this set
+ is empty and no DB query needs to fire.
+ """
+ if path_kind != "Dashboard":
+ return set()
+ return {
+ (record["entity_id"], record["transaction_id"])
+ for record in records
+ if TABLE_KIND_TO_API.get(record["entity_kind"]) == "SqlaTable"
+ }
+
+
+def batch_chart_counts(
+ dashboard_id: int, pairs: set[tuple[int, int]]
+) -> dict[tuple[int, int], int]:
+ """For every ``(dataset_id, target_tx)`` in *pairs*, count the
+ distinct charts that were both on *dashboard_id* and pointing at
+ *dataset_id* at *target_tx*.
+
+ One SELECT against ``dashboard_slices_version`` ⨝ ``slices_version``,
+ pulling the (slice, dataset, validity-window) state for every slice
+ ever on the dashboard whose dataset matches one of the requested
+ dataset_ids. The Python loop then applies the validity-strategy
+ predicate per pair. Replaces the previous N+1 shape that fired one
+ COUNT per related record.
+
+ Returns ``{(dataset_id, target_tx): count}``; pairs whose count
+ would be zero are omitted so the caller's ``.get(key, 0)`` is
+ correct.
+ """
+ if not pairs:
+ return {}
+
+ # pylint: disable=import-outside-toplevel
+ from sqlalchemy_continuum import version_class
+
+ from superset.models.slice import Slice
+
+ metadata = version_class(Slice).__table__.metadata
+ m2m_tbl = metadata.tables.get("dashboard_slices_version")
+ slices_tbl = version_class(Slice).__table__
+ if m2m_tbl is None:
+ return {}
+
+ dataset_ids = {dataset_id for dataset_id, _ in pairs}
+ # Bound both validity windows to the transaction range the page-set
+ # needs. Both the attachment (m2m) and the chart→dataset (slice) window
+ # must straddle a requested target_tx, so a row whose window starts
+ # after the newest target, or closes at/before the oldest one, can
+ # never contribute a match. Without this the join multiplies every
+ # attachment row by the full slice version history — an unbounded cross
+ # product on dashboards with long-lived, frequently-edited charts.
+ target_txs = {target_tx for _, target_tx in pairs}
Review Comment:
Addressed by subsequent commits; the flagged code is no longer current after the rebase onto master. Resolving as outdated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(versioning): cross-entity version activity view [superset]
Posted by "mikebridge (via GitHub)" <gi...@apache.org>.
mikebridge commented on PR #41076:
URL: https://github.com/apache/superset/pull/41076#issuecomment-4971280330
Follow-up on the review findings posted above: the blocking correctness items and associated cleanup were addressed in `0d6175e5bd`, `9a855a71c0`, and `ea43cd30a2`.
In particular, the updated implementation:
- globally merges bounded candidates across entity-ID chunks, preserving the newest records regardless of chunk order and reporting truncation consistently;
- bounds self and related history to the current `(id, uuid)` incarnation, preventing history leakage when integer IDs are reused;
- resolves historical UUIDs during decoration and redacts records that cannot be matched safely;
- handles degenerate shadow rows without turning an activity request into a 500;
- bounds name/impact lookups to the page transaction range and includes the smaller documentation/docstring cleanups;
- adds regression coverage for later-chunk newer records, current-incarnation clipping, empty untracked related scope, reused-ID redaction, and stable total ordering.
I also performed a fresh Docker/PostgreSQL dark-launch verification with `ENABLE_VERSIONING_CAPTURE` absent from the environment (resolved by Superset as `False`). I committed description changes to an existing chart, dashboard, and dataset and confirmed that the saves persisted while all capture tables remained empty:
- `slices_version`: 0
- `dashboards_version`: 0
- `tables_version`: 0
- `version_transaction`: 0
- `version_changes`: 0
Authenticated requests to each entity's `/activity/` endpoint also returned `count=0`, an empty `result`, and `truncated=false`. This confirms the feature remains inactive by default while the read endpoints continue to behave normally.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(versioning): cross-entity version activity view [superset]
Posted by "mikebridge (via GitHub)" <gi...@apache.org>.
mikebridge commented on PR #41076:
URL: https://github.com/apache/superset/pull/41076#issuecomment-4927469980
## Versioning base-infra — verified review notes (head `bf31b236`)
_Generated by a multi-lens review (committer / python / sqlalchemy / DDD lenses) and then independently verified against the source — every finding below was re-checked by reading the code at this head, and a few of the raw findings were dropped or down-scoped in that pass (noted where relevant). Line numbers are against `bf31b236`. SIP/process items are tracked separately and excluded here._
**TL;DR** — nothing here blocks the dark launch. It ships inert behind `ENABLE_VERSIONING_CAPTURE` (default off), the migration is additive/reversible, and the security posture (per-object `raise_for_access` on every read endpoint, fail-closed visibility filter, fail-open write path) is solid. The items below are (A) capture-path correctness bugs to fix **before** flipping the flag on, (B) small cleanups, and (C) forward-looking notes.
### A. Fix before enabling `ENABLE_VERSIONING_CAPTURE`
These are latent while capture is off, so they don't block merge — but they bite the moment capture is enabled.
**`superset/versioning/diff.py:425-430` — duplicate natural keys silently overwrite, dropping change records**
The by-key maps are built last-write-wins: two adhoc filters on the same `subject` column (or two metrics sharing a `label`) collapse to one, so editing/removing one emits a mangled single record and the other's change is lost. The `_filter_key` docstring's promise that same-column filters stay disambiguable can't hold once the collision drops one.
Would it be worth disambiguating on collision (secondary key, or fall back to positional index only for colliding items)?
**`superset/versioning/diff.py:420-427` (+ listener catch at `changes/listener.py:239`) — an unhashable `subject` raises `TypeError`, losing that entity's scalar diff**
`_filter_key` returns `f.get("subject")` (typed `Key`, but it's `Any` from user JSON). A list/dict subject flows unguarded into `from_by_key[<list>]` → `TypeError: unhashable`. It's caught per-entity, so the blast radius is just that one entity's scalar change records for the save (not the whole save) — but the record loss is silent. A `str(...)` coercion (or a fallback in `_effective_key` when the key isn't `str`/`int`/`None`) closes it.
**`superset/commands/importers/v1/utils.py:391-419` — `_prime_versioning_unit_of_work`'s broad `except` gives false assurance**
The docstring promises it "never breaks an import," but the broad `except Exception` only genuinely covers the ImportError (continuum-absent) case. If `unit_of_work()` fails for any other reason while capture is enabled, the very next `dashboard_slices.insert()` fires Continuum's `before_execute` with no UoW and raises `KeyError` — the import dies anyway, now with a misleading "proceeding without it" log. Narrowing to `except ImportError:` lets the real failure surface at its source. (Narrow trigger — `unit_of_work` rarely fails — but the false assurance is the issue.)
**`superset/versioning/activity/queries.py:444-489` — per-kind budget can drop later chunks' records and mislabel the window as complete**
When one kind's entities span multiple id-chunks and fill the ~5000-row budget, the loop `break`s and later chunks contribute zero rows — including records newer than the truncation floor — yet `truncated=True` presents it as a clean "older records omitted" cut. (Correction to the raw finding: the drop order is arbitrary but **deterministic** — not `PYTHONHASHSEED`-dependent — since these are `set[int]`. High bar: >500 entities of one kind **and** >5000 matching rows.) Applying the ceiling after concatenating + re-sorting all chunks (and `sorted(entity_ids)` before chunking) makes any residual drop deterministic and correctly labeled.
### B. Small cleanups (unconditional)
**`superset/versioning/diff.py:799-805` — `_meta_excluding_position` doesn't exclude anything**
Name + docstring promise positional bits are stripped from `meta`; the body just returns `dict(meta)` unchanged. It's correct-by-accident today (position lives on `node["parents"]`, handled separately), but the next field added to `meta` silently breaks the move-vs-edit contract the name advertises. Rename to `_meta_copy` (+ honest docstring) or actually implement the strip.
**`Slice` → `chart` dispatch is spread across ~8 parallel structures keyed on the class-name string**
`ENTITY_KIND_BY_CLASS_NAME` (`changes/table.py:86`), `TABLE_KIND_TO_API` / `API_KIND_TO_TABLE` / `API_KIND_LABEL` / `USER_FACING_KIND` / `NOT_FOUND_EXC` / `NAME_COLUMN` (`activity/kinds.py:73-115`), `_RAISE_FOR_ACCESS_KWARG` (`api_helpers.py:136`), plus string-dispatch ladders in `changes/state.py`. Adding a fourth versioned entity means editing all of them in lock-step with nothing catching a miss. A single registry (one entry per entity, constructor-checked for completeness) turns "did I update every table?" into a type error. The DDD lens flagged this as the one durable must-fix as the surface grows.
**`superset/daos/dataset.py:534-537` & `:602-605` — `_upsert_columns` / `update_metrics` lack the `protected_keys` guard**
The sibling `_override_columns` (`:512-516`) protects `("id", "table_id")` before `setattr`; these two loops set every payload key unguarded. Benign today, but it's the exact footgun the sibling was hardened against — worth the same guard for symmetry.
### C. Forward-looking (not this PR, but worth a note)
- **Shadow-table FKs to `version_transaction` have no `ondelete`, while `version_changes.transaction_id` has `ondelete="CASCADE"`** (migration `56cd24c07170`). When the retention task (stacked PR) deletes `version_transaction` rows, it must delete referencing shadow rows first, in order, or every prune cycle hits an FK violation on PG/MySQL. Either add `ondelete="CASCADE"` to the six shadow FKs, or make the deletion-ordering contract explicit where retention lands.
- **Migration idempotency is inconsistent** — `add_versioning_tables` uses raw `create_table` (a partial failure on MySQL's non-transactional DDL isn't re-runnable), while the index follow-up migration is inspector-guarded. Worth matching the guarded pattern (or documenting the manual-cleanup caveat).
- **Per-save cost once capture is on** — `SkipUnmodifiedPlugin._matches_previous_version` (`factory.py:294`) + `_read_pre_state` add two full-row reads (incl. large `MediumText` columns) per versioned save. `perf_validation_tests.py` is the right place to surface numbers before the default flips.
### Checked and cleared
- `SqlaTable.__versioned__` excluding `editors` but not `viewers` — **not a problem**: `SqlaTable` has no `viewers` relationship, so there's nothing to exclude (unlike Dashboard/Slice).
- Import-time `make_versioned()` in `superset/extensions/__init__.py` — no model classes are imported at that point, so it's the correct Continuum ordering, not an import-safety regression.
Praise where due: the fail-open write path + dedicated capture-error metrics, the thorough kill-switch (detaches Continuum's own listeners *and* verifies), the deterministic UUIDv5 version identity, the pure/functional diff engine with `ChangeRecord`/`Window` as clean value objects, and an unusually honest `UPDATING.md` are all genuinely well done.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(versioning): cross-entity version activity view [superset]
Posted by "netlify[bot] (via GitHub)" <gi...@apache.org>.
netlify[bot] commented on PR #41076:
URL: https://github.com/apache/superset/pull/41076#issuecomment-4712528723
### <span aria-hidden="true">✅</span> Deploy Preview for *superset-docs-preview* ready!
| Name | Link |
|:-:|------------------------|
|<span aria-hidden="true">🔨</span> Latest commit | d6d99aa661728a94e88d717b3835cf993c2f66b9 |
|<span aria-hidden="true">🔍</span> Latest deploy log | https://app.netlify.com/projects/superset-docs-preview/deploys/6a306b9b06a1d90008426807 |
|<span aria-hidden="true">😎</span> Deploy Preview | [https://deploy-preview-41076--superset-docs-preview.netlify.app](https://deploy-preview-41076--superset-docs-preview.netlify.app) |
|<span aria-hidden="true">📱</span> Preview on mobile | <details><summary> Toggle QR Code... </summary><br /><br /><br /><br />_Use your smartphone camera to open QR code link._</details> |
|<span aria-hidden="true">🤖</span> Make changes | [Run an agent on this branch](https://app.netlify.com/projects/superset-docs-preview/agent-runs#DcpLCoAgEADQu8w6Ycow6wYdwx81UCoqhkR3z91bvBdiCncssAEMoJPy5uzOho24TJKz6lKm4MkfTJlClUpjldzTt3XxCm23_QvFUehVo1CjXRFRzpOQuMD3Aw) |
---
<!-- [superset-docs-preview Preview](https://deploy-preview-41076--superset-docs-preview.netlify.app) -->
_To edit notification comments on pull requests, go to your [Netlify project configuration](https://app.netlify.com/projects/superset-docs-preview/configuration/notifications#deploy-notifications)._
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(versioning): cross-entity version activity view [superset]
Posted by "mikebridge (via GitHub)" <gi...@apache.org>.
mikebridge commented on code in PR #41076:
URL: https://github.com/apache/superset/pull/41076#discussion_r3574958428
##########
superset/daos/version.py:
##########
@@ -0,0 +1,69 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Backward-compat façade for the entity-versioning DAO surface.
+
+The actual implementation lives in :mod:`superset.versioning.queries`
+(read side: list/get/resolve/find/UUID derivation). This module
+re-exports it under a single ``VersionDAO`` class plus the module-level
+UUID helpers so existing callers keep working without changes. (The
+write side — restore + audit stamping — ships in a later PR; only the
+read surface is wired here.)
+
+New code should import from the versioning sub-modules directly.
+"""
+
+from __future__ import annotations
+
+from superset.versioning.queries import (
+ current_live_transaction_id,
+ current_live_version_uuid,
+ current_version_number,
+ derive_version_uuid,
+ derive_version_uuid as _derive_version_uuid, # noqa: F401
+ find_active_by_uuid,
+ get_version,
+ list_change_records_batch,
+ list_versions,
+ resolve_version_uuid,
+ VERSION_UUID_NAMESPACE,
+)
+
+# Re-exports for ``from superset.daos.version import …`` consumers.
+__all__ = [
+ "VERSION_UUID_NAMESPACE",
+ "VersionDAO",
+ "derive_version_uuid",
+]
Review Comment:
Addressed by subsequent commits; the flagged code is no longer current after the rebase onto master. Resolving as outdated.
##########
superset/datasets/api.py:
##########
@@ -440,17 +504,69 @@ def put(self, pk: int) -> Response:
# This validates custom Schema with custom validations
except ValidationError as error:
return self.response_400(message=error.messages)
+
+ # Live version identifiers before the update (empty + query-free when
+ # ``ENABLE_VERSIONING_CAPTURE`` is off).
+ old_info = current_entity_version_info(SqlaTable, pk)
Review Comment:
Addressed by subsequent commits; the flagged code is no longer current after the rebase onto master. Resolving as outdated.
##########
superset/datasets/api.py:
##########
@@ -440,17 +504,69 @@ def put(self, pk: int) -> Response:
# This validates custom Schema with custom validations
except ValidationError as error:
return self.response_400(message=error.messages)
+
+ # Live version identifiers before the update (empty + query-free when
+ # ``ENABLE_VERSIONING_CAPTURE`` is off).
+ old_info = current_entity_version_info(SqlaTable, pk)
+
try:
+ # Two commands, two commits, two Continuum transactions for an
+ # ``override_columns`` save — deliberately NOT merged into one
+ # transaction. A single-transaction design was attempted and
+ # reverted: ``DBEventLogger`` writes request logs through the
+ # SHARED scoped session and calls ``commit()`` /
+ # ``rollback()`` on it mid-request (superset/utils/log.py),
+ # so any save held uncommitted across a logged sub-action can
+ # be committed half-done (Postgres/MySQL) or rolled back
+ # entirely on a transient logger failure (SQLite's
+ # "database is locked"). Until the event logger gets its own
+ # session, per-command commit boundaries are the only shape
+ # whose failure modes are honest. Consequence the
+ # version-history UI must tolerate: one logical save can
+ # surface as two version transactions stamped the same second.
changed_model = UpdateDatasetCommand(pk, item, override_columns).run()
+ # Capture the post-update identifiers BEFORE the refresh:
+ # RefreshDatasetCommand commits its own transaction, so reading
+ # afterwards would attribute the refresh's version to the
+ # user's update (and old→new would span two transactions).
+ new_info = current_entity_version_info(
+ SqlaTable, changed_model.id, changed_model.uuid
+ )
Review Comment:
Addressed by subsequent commits; the flagged code is no longer current after the rebase onto master. Resolving as outdated.
##########
superset/datasets/api.py:
##########
@@ -440,17 +504,69 @@ def put(self, pk: int) -> Response:
# This validates custom Schema with custom validations
except ValidationError as error:
return self.response_400(message=error.messages)
+
+ # Live version identifiers before the update (empty + query-free when
+ # ``ENABLE_VERSIONING_CAPTURE`` is off).
+ old_info = current_entity_version_info(SqlaTable, pk)
+
try:
+ # Two commands, two commits, two Continuum transactions for an
+ # ``override_columns`` save — deliberately NOT merged into one
+ # transaction. A single-transaction design was attempted and
+ # reverted: ``DBEventLogger`` writes request logs through the
+ # SHARED scoped session and calls ``commit()`` /
+ # ``rollback()`` on it mid-request (superset/utils/log.py),
+ # so any save held uncommitted across a logged sub-action can
+ # be committed half-done (Postgres/MySQL) or rolled back
+ # entirely on a transient logger failure (SQLite's
+ # "database is locked"). Until the event logger gets its own
+ # session, per-command commit boundaries are the only shape
+ # whose failure modes are honest. Consequence the
+ # version-history UI must tolerate: one logical save can
+ # surface as two version transactions stamped the same second.
changed_model = UpdateDatasetCommand(pk, item, override_columns).run()
+ # Capture the post-update identifiers BEFORE the refresh:
+ # RefreshDatasetCommand commits its own transaction, so reading
+ # afterwards would attribute the refresh's version to the
+ # user's update (and old→new would span two transactions).
+ new_info = current_entity_version_info(
+ SqlaTable, changed_model.id, changed_model.uuid
+ )
+ etag_version_uuid = new_info.version_uuid
Review Comment:
Addressed by subsequent commits; the flagged code is no longer current after the rebase onto master. Resolving as outdated.
##########
scripts/seed_junction_load.py:
##########
@@ -0,0 +1,679 @@
+#!/usr/bin/env python3
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+# ----------------------------------------------------------------------
+# Stress-test data generator for the composite-PK migration (sc-105349).
+#
+# Bulk-inserts synthetic parent rows and many-to-many junction rows for
+# the eight association tables that the composite-PK migration touches.
+# Useful for measuring migration runtime at varying scales — run this at
+# 100K / 1M / 5M / 10M rows and time the migration at each scale to
+# verify the O(N log N) extrapolation.
+#
+# Idempotent: rerunning with the same target is a no-op; rerunning with
+# a higher target adds rows up to the new total. Batched bulk INSERTs
+# (10K rows per statement) make it fast on Postgres, MySQL, and SQLite.
+#
+# Usage (inside the Superset container):
+#
+# docker exec superset-superset-1 \\
+# /app/.venv/bin/python /app/scripts/seed_junction_load.py \\
+# --dashboard-slices 1000000 \\
+# --slice-user 100000 \\
+# --dashboard-user 100000
+#
+# Run with no flags for the defaults shown below. Use ``--dry-run`` to
+# print the planned inserts without writing anything.
+#
+# The script connects via Superset's standard ``DATABASE_*`` env vars
+# (or ``SUPERSET__SQLALCHEMY_DATABASE_URI`` if set), so it works
+# automatically inside the Superset container regardless of which
+# metadata DB backend is in use.
+
+from __future__ import annotations
+
+import argparse
+import logging
+import os
+import sys
+import time
+from contextlib import contextmanager
+from typing import Iterator
+from uuid import uuid4
+
+import sqlalchemy as sa
+from sqlalchemy.engine import Connection, Engine
+
+logger = logging.getLogger("seed_junction_load")
Review Comment:
Addressed by subsequent commits; the flagged code is no longer current after the rebase onto master. Resolving as outdated.
##########
scripts/seed_junction_load.py:
##########
@@ -0,0 +1,679 @@
+#!/usr/bin/env python3
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+# ----------------------------------------------------------------------
+# Stress-test data generator for the composite-PK migration (sc-105349).
+#
+# Bulk-inserts synthetic parent rows and many-to-many junction rows for
+# the eight association tables that the composite-PK migration touches.
+# Useful for measuring migration runtime at varying scales — run this at
+# 100K / 1M / 5M / 10M rows and time the migration at each scale to
+# verify the O(N log N) extrapolation.
+#
+# Idempotent: rerunning with the same target is a no-op; rerunning with
+# a higher target adds rows up to the new total. Batched bulk INSERTs
+# (10K rows per statement) make it fast on Postgres, MySQL, and SQLite.
+#
+# Usage (inside the Superset container):
+#
+# docker exec superset-superset-1 \\
+# /app/.venv/bin/python /app/scripts/seed_junction_load.py \\
+# --dashboard-slices 1000000 \\
+# --slice-user 100000 \\
+# --dashboard-user 100000
+#
+# Run with no flags for the defaults shown below. Use ``--dry-run`` to
+# print the planned inserts without writing anything.
+#
+# The script connects via Superset's standard ``DATABASE_*`` env vars
+# (or ``SUPERSET__SQLALCHEMY_DATABASE_URI`` if set), so it works
+# automatically inside the Superset container regardless of which
+# metadata DB backend is in use.
+
+from __future__ import annotations
+
+import argparse
+import logging
+import os
+import sys
+import time
+from contextlib import contextmanager
+from typing import Iterator
+from uuid import uuid4
+
+import sqlalchemy as sa
+from sqlalchemy.engine import Connection, Engine
+
+logger = logging.getLogger("seed_junction_load")
+
+# Bulk INSERT batch size. Larger values = fewer statements but more memory.
+BATCH = 10_000
Review Comment:
Addressed by subsequent commits; the flagged code is no longer current after the rebase onto master. Resolving as outdated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org