You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@superset.apache.org by "mikebridge (via GitHub)" <gi...@apache.org> on 2026/08/24 21:18:31 UTC

[PR] feat(deletion-retention): scheduled pruning policy for the purge audit log [superset]

mikebridge opened a new pull request, #43490:
URL: https://github.com/apache/superset/pull/43490

   ### SUMMARY
   
   The `purge_audit_log` table grew without bound. Every scheduled purge run that hit the same blocked entity wrote another identical record (until #42863 deduped the write path), and nothing ever aged out operational noise. This adds the retention policy defined by [SC-116701](https://app.shortcut.com/preset/story/116701) and the scheduled task that applies it. Deletion-only — no schema change, no migration.
   
   Three categories, drained in priority order under one shared per-run budget:
   
   1. **Blocked duplicates.** Within an entity's *current* blockage streak, only the earliest row survives — it carries the "blocked since" fact and is never deleted while the streak is current. Later duplicates go regardless of age; that is the rule that actually bounds growth.
   2. **Operational expiry.** `failed` rows and `blocked` rows from *resolved* streaks age out past `PURGE_AUDIT_OPERATIONAL_RETENTION_DAYS` (default 90).
   3. **Evidence expiry.** `confirmed` / `target_absent` — the only surviving trace of a destroyed object — are **never** touched unless `PURGE_AUDIT_EVIDENCE_RETENTION_DAYS` is explicitly set. Setting it is the operator's assertion that an approved compliance policy permits expiring destruction evidence.
   
   `pending` rows belong to `reconcile_pending()` and are structurally unreachable. Invalid retention values fail closed: the run logs a warning and skips that category rather than widening removal. `PURGE_AUDIT_PRUNING_ENABLED = False` restores the previous behaviour exactly, and a disabled run says so rather than silently doing nothing.
   
   **Design notes worth a reviewer's attention:**
   
   - **A `failed` attempt does not end a blockage streak.** A failed purge is an infrastructure outcome — the cascade raised — not evidence the block cleared; the blocking policy is untouched. Treating it as a boundary would let one transient error demote the blocked-since survivor and restate the blockage as beginning *after* the failure.
   - **Concurrency.** Deletes are conditional on the expected status and counted from statement rowcounts, so overlapping runs cannot double-remove or double-report. Beyond that, a blocked row whose streak classification is *unstable* is never deleted by either category: finalizing a `pending` row resolves it in place, keeping its original timestamp, so an unresolved attempt is a boundary that can appear mid-history and turn the row after it into a survivor. Evidence expiry likewise refuses to delete a row that still bounds surviving blocked rows, which is what stops a boundary receding — and makes an evidence window shorter than the operational one safe rather than corrupting.
   - **Known limitation, documented in the module docstring.** These guards are evaluated when candidates are selected, not when they are deleted. `created_on` is stamped by the writing process rather than the database, so a row can become visible with a timestamp in the past; one appearing in the window between the SELECT and the DELETE would evade them. Closing it completely needs candidacy evaluated *inside* the DELETE (dialect-specific: MySQL rejects self-referencing DELETE subqueries), serialization against the audit writers, or a database-assigned `created_on`. Tracked as follow-up; the exposure is bounded by writer clock spread and by how closely spaced an entity's blocked rows are.
   - Each category is guaranteed at least one batch of the shared budget. Strict priority would let a permanent duplicate backlog starve age-out forever — reintroducing the unbounded growth this feature exists to stop.
   - The startup warning that already covers the other retention tasks now covers this beat entry too, so an operator who replaced `CELERY_CONFIG` is told pruning is not running.
   
   This PR was reviewed by a nine-lens panel plus three adversarial data-systems passes before opening; the survivor-invariant findings from those passes are fixed here, each with a red-first control run.
   
   ### BEFORE/AFTER SCREENSHOTS OR ANIMATED GIF
   
   No UI. Operator-visible surface is configuration, metrics (`deletion_retention.prune_purge_audit.*` — per-category removal counts plus `carried_over`), and one structured completion log line per run.
   
   ### TESTING INSTRUCTIONS
   
   ```bash
   # Unit
   pytest tests/unit_tests/commands/deletion_retention/test_prune_audit.py
   
   # Integration (needs a metadata DB; run against Postgres)
   pytest tests/integration_tests/deletion_retention/prune_audit_tests.py
   ```
   
   26 unit tests (classification and the exhaustiveness contract against the model's `ALL_STATUSES`, fail-closed config validation, task reporting/isolation, clock parity with the audit writer) and 20 integration tests against the real table: age-independent dedup, bounded convergence with carryover, cross-entity isolation, resolved streaks, `failed` streak-transparency, evidence protection by count-and-identity, opt-in expiry and its disable-again edge, pending/future-row immunity, skew-resistant classification, unstable-block deferral in both categories, boundary-guard deferral including the inverted-window case, uuid-less block retention, starvation resistance, rerun idempotence, and the status re-check.
   
   Manual: enable the beat entry, seed duplicate `blocked` rows for one entity, run `deletion_retention.prune_purge_audit`, and confirm the earliest row survives and the counts appear in logs/metrics. Full recipe in `specs/sc-116701-purge-audit-pruning/quickstart.md`.
   
   ### ADDITIONAL INFORMATION
   - [x] Has associated issue: SC-116701 (follow-up to SC-115343 / #42863, part of the soft-delete work under the approved [SIP-208](https://github.com/apache/superset/issues/39286))
   - [x] Required feature flags: none — pruning is independent of `SOFT_DELETE` because audit rows outlive the flag. The beat-entry startup warning is gated on it.
   - [ ] Changes UI
   - [ ] Includes DB Migration (follow approval process in [SIP-59](https://github.com/apache/superset/issues/13351))
     - [ ] Migration is atomic, supports rollback & is backwards-compatible
     - [ ] Confirm DB migration upgrade and downgrade tested
     - [ ] Runtime estimates and downtime expectations provided
   - [x] Introduces new feature or API — new Celery task, three config keys, no REST surface
   - [ ] Removes existing feature or API
   
   **Behaviour change on upgrade** (recorded in `UPDATING.md`): operational audit records older than 90 days begin pruning automatically. Completed-destruction evidence is untouched. Opt out with `PURGE_AUDIT_PRUNING_ENABLED = False`; deployments overriding `CELERY_CONFIG` must carry the new beat entry forward.
   
   This PR was developed with AI assistance (Claude Code), including the implementation, tests, and review remediation; a human (@mikebridge) reviews before merge.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org


Re: [PR] feat(deletion-retention): scheduled pruning policy for the purge audit log [superset]

Posted by "mikebridge (via GitHub)" <gi...@apache.org>.
mikebridge commented on PR #43490:
URL: https://github.com/apache/superset/pull/43490#issuecomment-5535100528

   @aminghadersohi — thanks, Finding 1 is a fair call. Addressed at `e2dddd9da7`.
   
   **On the lock scope itself — kept deliberately, not narrowed.** You praised the lock-then-clock ordering, and that's exactly why I didn't narrow it: holding the lock across the DELETE is what serializes a pruning batch with `write_ahead`'s row so a concurrent write commits only *after* the batch (invisible to its SELECT). A "timestamp-only prelude" narrowing would reintroduce the race, and I can't prove a safe narrowing correct without concurrency test coverage the environment here can't run. So per the tradeoff you flagged, the window is bounded instead — `BATCH_SIZE=500` caps rows per statement and the pruning indexes back the boundary/per-entity subqueries — and the liveness cost is documented rather than papered over.
   
   **What changed (docs only, no logic):**
   - `write_ahead`: added the load-bearing comment you asked for on why the lock precedes `created_on=utc_now()`.
   - `_delete_batch`: comment on why the lock spans the DELETE + the bounded liveness tradeoff.
   - `UPDATING.md`: the operator-facing note — a concurrent purge's write-ahead can block until the batch commits; it's fail-closed-and-retried where a lock/statement timeout applies, so a skipped cycle, not data loss.
   
   **Two accuracy corrections a self-review of these comments surfaced, worth flagging since they touch your finding:**
   1. The guarantee is transaction **visibility** (isolation), not a wall-clock one. A pure `created_on > now` claim only holds under bounded clock skew (`write_ahead` and the pruner are separate tasks with independent clocks — and the module already guards writer skew via the future-row exclusion in `_streak_boundary_subquery`). The comments now lead with visibility and name the streak invariants, not wall-clock ordering, as the primary safeguard.
   2. "Fails closed on the lock-wait timeout" isn't true for default PostgreSQL (`lock_timeout` disabled → it waits then succeeds). The docs now spell out the dialect split (PG waits-then-succeeds; MySQL `innodb_lock_wait_timeout` / SQLite / a configured timeout fail closed).
   
   Full 4-lens pass (clean-code/tidy-first/python/sqlalchemy) on the change is filed in the spec repo. The rebase (single alembic head at `8f31c5d726ab`), the stale-approval note, and the earlier delta-review fix are all still in place. Would appreciate another look.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org


Re: [PR] feat(deletion-retention): scheduled pruning policy for the purge audit log [superset]

Posted by "mikebridge (via GitHub)" <gi...@apache.org>.
mikebridge commented on code in PR #43490:
URL: https://github.com/apache/superset/pull/43490#discussion_r3978175795


##########
UPDATING.md:
##########
@@ -945,7 +980,7 @@ With the flag on, delete confirmations across the chart/dashboard/dataset list p
 
 This also resolves the limitation noted under *Soft delete and restore for datasets*: a database blocked by soft-deleted datasets can now be freed by purging those datasets (per-entity endpoint, retention task, or `force-purge` CLI) instead of hard-deleting `tables` rows out-of-band.
 
-The `purge_audit_log` table is **never pruned by design** — the audit must survive the entities it names; operators who need to age it out should prune manually.
+The `purge_audit_log` table is pruned automatically by the `deletion_retention.prune_purge_audit` Celery beat task (daily, 03:30), so it no longer grows unbounded and does not need manual pruning. The policy is written to preserve the audit's meaning rather than trade it away: within an entity's current blockage streak the earliest — "blocked since" — record always survives (only redundant duplicate `blocked` records are collapsed), and completed-destruction evidence (`confirmed`, `target_absent`) is **never** removed unless the separate `PURGE_AUDIT_EVIDENCE_RETENTION_DAYS` opt-in is explicitly set. What ages out is operational noise — `blocked` records from already-resolved streaks and `failed` records — once older than `PURGE_AUDIT_RETENTION_DAYS` (default 90). Set `PURGE_AUDIT_PRUNING_ENABLED = False` to restore the previous never-pruned, unbounded-growth behavior. See the release-note entry above for the beat-schedule and `CELERY_CONFIG` details.

Review Comment:
   Fixed in f933a93801. You are right on both counts: `PURGE_AUDIT_RETENTION_DAYS` is read nowhere (the key the code reads is `PURGE_AUDIT_OPERATIONAL_RETENTION_DAYS`, prune_audit.py:149 / config.py:1049), and pruning ships **off** (`PURGE_AUDIT_PRUNING_ENABLED = False`, config.py:1040). Rewrote this UPDATING.md entry to say pruning is off by default and must be enabled explicitly with `PURGE_AUDIT_PRUNING_ENABLED = True`, and corrected the retention key — so it no longer reads as bounded-by-default. (The release-note entry higher in the file was already correct; this detailed paragraph was the one that drifted.)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org


Re: [PR] feat(deletion-retention): scheduled pruning policy for the purge audit log [superset]

Posted by "mikebridge (via GitHub)" <gi...@apache.org>.
mikebridge commented on PR #43490:
URL: https://github.com/apache/superset/pull/43490#issuecomment-5637558385

   @sadpandajoe — thanks for the careful reviews. The two older threads (force-purge exemption, the UPDATING key/default) are resolved.
   
   On the lock-hold concern (`prune_audit.py:436`): reworked in `ad34aeee0f`. Discovery now runs **unlocked** (a ≤500-id hint), then under the coordination lock a re-check re-applies the same candidacy predicates and deletes survivors by literal id — so the locked work is bounded to per-PK + per-entity index probes, not an O(blocked-rows) scan. The streak boundary is now a correlated per-entity scalar shared by both discovery and re-check (one predicate list, so they can't drift), and the backdated-write invariant holds — a row a concurrent `write_ahead` turns into a survivor between discovery and the lock fails the re-check and isn't deleted. Added a two-session test that commits a pending row in the discovery→lock window under **REPEATABLE READ** (fails without the fix, passes with it).
   
   Mind taking another look? Happy to walk through any of it.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org


Re: [PR] feat(deletion-retention): scheduled pruning policy for the purge audit log [superset]

Posted by "rusackas (via GitHub)" <gi...@apache.org>.
rusackas commented on code in PR #43490:
URL: https://github.com/apache/superset/pull/43490#discussion_r3864766044


##########
superset/commands/deletion_retention/prune_audit.py:
##########
@@ -0,0 +1,607 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Deletion-only pruning of the ``purge_audit_log`` table.
+
+Bounds the audit history's growth without ever weakening its evidentiary
+value. Three delete categories, applied in priority order under one shared
+per-run batch budget:
+
+1. **Blocked duplicates** — within an entity's *current* blockage streak
+   (its ``blocked`` rows newer than the entity's newest streak-breaking
+   row), only the streak's earliest row survives: later duplicates are
+   removed regardless of age. The survivor carries the "blocked since"
+   fact and is never deleted while the streak is current.
+2. **Operational expiry** — ``blocked`` rows of *resolved* streaks and
+   ``failed`` rows older than ``PURGE_AUDIT_OPERATIONAL_RETENTION_DAYS``
+   age out.
+3. **Evidence expiry** — ``confirmed`` / ``target_absent`` rows are
+   untouchable unless ``PURGE_AUDIT_EVIDENCE_RETENTION_DAYS`` is
+   explicitly set (the operator's compliance assertion), and then only
+   rows older than that window.
+
+``pending`` rows belong to :func:`audit.reconcile_pending` and rows with
+future timestamps (clock skew) are excluded from every category *and* from
+streak classification, so a skewed writer cannot reclassify a live streak.
+
+Three invariants keep the survivor safe without a distributed lock, which
+matters because runs can overlap and both ``reconcile_pending`` and the
+purge path finalize rows concurrently. A duplicate is only ever deleted
+when no concurrent transition could turn it into a survivor first:
+
+* **A boundary is never removed while it still bounds anything.** Evidence
+  expiry refuses to delete a row while an older ``blocked`` or ``pending``
+  row for the same entity survives. Boundaries therefore never *recede*,
+  which would otherwise promote resolved rows into a current streak.
+* **A blocked row whose classification is unstable is never deleted.**
+  Finalizing a ``pending`` row resolves it *in place*, keeping its original
+  timestamp, so an unresolved attempt is a boundary that may appear
+  mid-history at any moment — and the blocked row after it would become its
+  streak's survivor. Blocked rows preceded by an unresolved attempt are
+  therefore skipped by **both** the duplicate and the operational category.
+  Age does not stabilize the classification: an old blocked row inside a
+  live streak is exactly the "blocked for years" case FR-009 protects.
+  Without this, a boundary moving *forward* would demote the current
+  survivor and promote the next row into its place — possibly a row already
+  selected for deletion.
+* **Deletes are conditional and counted from rowcounts.** A row whose
+  status changed since selection is not matched, so overlapping runs can
+  neither double-remove nor double-report.
+
+Boundaries moving forward past *all* of an entity's blocked rows leave no
+survivor to promote, so a selected duplicate may be removed slightly ahead
+of its retention window in that case. It was redundant either way and the
+streak's earliest row is untouched.
+
+Candidate selection is embedded in each ``DELETE`` statement. The derived-table
+wrapper keeps that shape legal on MySQL while ensuring that a pending or
+recovered row committed before the delete is evaluated participates in the
+survivor and boundary predicates. No stale list of candidate ids crosses a
+transaction boundary.
+
+Audit creation/recovery and every pruning batch take the same singleton
+database write lock before assigning a timestamp or evaluating candidates.
+The lock is held through commit, so an audit row cannot become visible in an
+already-processed logical past. Automatic pruning still ships disabled by
+default so operators explicitly choose their retention policy.
+"""
+
+from __future__ import annotations
+
+import logging
+from collections.abc import Callable
+from dataclasses import dataclass, field
+from datetime import datetime, timedelta
+from functools import partial
+from typing import Any, Literal, NamedTuple, TypeAlias
+
+import sqlalchemy as sa
+from flask import current_app
+
+from superset import db
+from superset.commands.deletion_retention.audit import (
+    acquire_coordination_lock,
+    utc_now,
+)
+from superset.models.purge_audit_log import (
+    PurgeAuditLog,
+    STATUS_BLOCKED,
+    STATUS_CONFIRMED,
+    STATUS_FAILED,
+    STATUS_PENDING,
+    STATUS_TARGET_ABSENT,
+)
+
+logger: logging.Logger = logging.getLogger(__name__)
+
+#: Operational records: noise-prone outcomes whose compliance value decays —
+#: a blocked or failed purge leaves the object in place (FR-001).
+OPERATIONAL_STATUSES: frozenset[str] = frozenset({STATUS_BLOCKED, STATUS_FAILED})
+#: Protected evidence: the only surviving trace of a destroyed object.
+PROTECTED_STATUSES: frozenset[str] = frozenset({STATUS_CONFIRMED, STATUS_TARGET_ABSENT})
+#: The outcomes that end a blockage streak — proof the object is gone.
+#:
+#: ``failed`` is deliberately absent. A failed purge is an infrastructure
+#: outcome (the cascade raised), not evidence the blockage cleared: the
+#: policy that blocked the entity is untouched, so the blockage continues
+#: across it. Treating ``failed`` as a boundary would let one transient
+#: error demote the "blocked since" survivor to an ageing duplicate and
+#: restate the blockage as beginning after the failure — losing exactly the
+#: fact FR-003 exists to preserve. ``pending`` is provisional and likewise
+#: neither joins nor breaks streaks.
+_STREAK_BREAKING_STATUSES: frozenset[str] = frozenset(
+    {STATUS_CONFIRMED, STATUS_TARGET_ABSENT}
+)
+
+#: Rows deleted per statement, matching the purge task's batch convention.
+BATCH_SIZE: int = 500
+#: One shared budget for the whole run across all three categories; the
+#: remaining backlog carries over to the next scheduled run (FR-004/SC-004).
+MAX_BATCHES_PER_RUN: int = 10
+
+OPERATIONAL_RETENTION_KEY: str = "PURGE_AUDIT_OPERATIONAL_RETENTION_DAYS"
+EVIDENCE_RETENTION_KEY: str = "PURGE_AUDIT_EVIDENCE_RETENTION_DAYS"
+
+
+class ResolvedWindow(NamedTuple):
+    """A retention window resolved from config.
+
+    Distinguishes the three outcomes a caller must tell apart: a usable
+    window, the deliberate "off" default, and operator error. Collapsing
+    the last two into a bare ``None`` would force every caller to re-read
+    config to find out which it got.
+    """
+
+    days: int | None
+    invalid_key: str | None = None
+
+
+def _validated_window(key: str, value: Any) -> ResolvedWindow:
+    """Validate a configured day count, failing closed on anything odd.
+
+    An invalid value disables its category for the run rather than widening
+    removal (FR-005/SC-005).
+    """
+    # bool is an int subclass and floats would silently truncate — both are
+    # config mistakes, not day counts, on a knob that deletes rows.
+    if not isinstance(value, bool) and not isinstance(value, float):
+        try:
+            days: int = int(value)
+        except (TypeError, ValueError):
+            days = 0
+        if days > 0:
+            return ResolvedWindow(days)
+    logger.warning(
+        "prune_audit: invalid %s=%r; skipping this category for the run "
+        "(pruning never widens on bad configuration)",
+        key,
+        value,
+    )
+    return ResolvedWindow(None, key)
+
+
+def resolve_operational_retention_days() -> ResolvedWindow:
+    """The operational retention window, or a disabled window when invalid."""
+    return _validated_window(
+        OPERATIONAL_RETENTION_KEY, current_app.config.get(OPERATIONAL_RETENTION_KEY)
+    )
+
+
+def resolve_evidence_retention_days() -> ResolvedWindow:
+    """The evidence expiration window; disabled unless explicitly opted in.
+
+    Unset is the documented "never expire evidence" default (FR-006), not an
+    error, so it produces a disabled window with no warning.
+    """
+    value: Any = current_app.config.get(EVIDENCE_RETENTION_KEY)
+    if value is None:
+        return ResolvedWindow(None)
+    return _validated_window(EVIDENCE_RETENTION_KEY, value)
+
+
+@dataclass
+class PruneRunResult:
+    """Per-category removal counts and run disposition for one pruning run."""
+
+    blocked_duplicates: int = 0
+    operational_expired: int = 0
+    evidence_expired: int = 0
+    #: True when the shared batch budget ran out before every category's
+    #: candidates were drained; the remainder converges on later runs.
+    carried_over: bool = False
+    invalid_config_keys: list[str] = field(default_factory=list)
+
+    @property
+    def total_removed(self) -> int:
+        """Total rows removed across every category."""
+        return (
+            self.blocked_duplicates + self.operational_expired + self.evidence_expired
+        )
+
+    def as_dict(self) -> dict[str, Any]:
+        """The task-return / log-line shape of this result."""
+        return {
+            "removed": {
+                "blocked_duplicates": self.blocked_duplicates,
+                "operational_expired": self.operational_expired,
+                "evidence_expired": self.evidence_expired,
+            },
+            "carried_over": self.carried_over,
+            "invalid_config_keys": list(self.invalid_config_keys),
+        }
+
+
+def _streak_boundary_subquery(now: datetime) -> sa.Subquery:
+    """Per entity, the ``created_on`` of its newest streak-breaking row.
+
+    Entities absent from this subquery have never been proven destroyed —
+    all their blocked rows form one current streak. Future-dated rows are
+    excluded so a skewed writer clock cannot push the boundary ahead of a
+    live streak and make its rows look resolved.
+    """
+    table: sa.Table = PurgeAuditLog.__table__
+    return (
+        sa.select(
+            table.c.entity_type.label("entity_type"),
+            table.c.entity_uuid.label("entity_uuid"),
+            sa.func.max(table.c.created_on).label("boundary"),
+        )
+        .where(table.c.status.in_(_STREAK_BREAKING_STATUSES))
+        .where(table.c.entity_uuid.is_not(None))
+        .where(table.c.created_on <= now)
+        .group_by(table.c.entity_type, table.c.entity_uuid)
+        .subquery("streak_boundary")
+    )
+
+
+def _survivor_subquery(now: datetime) -> sa.Subquery:
+    """Per entity, the ``created_on`` of its current streak's earliest row.
+
+    The survivor is the earliest ``blocked`` row strictly newer than the
+    entity's boundary (or its earliest blocked row outright when no boundary
+    exists). Rows sharing that exact timestamp are all treated as survivors —
+    with microsecond precision ties are pathological, and keeping an extra
+    row errs on the preserving side.
+    """
+    table: sa.Table = PurgeAuditLog.__table__
+    boundary: sa.Subquery = _streak_boundary_subquery(now)
+    return (
+        sa.select(
+            table.c.entity_type.label("entity_type"),
+            table.c.entity_uuid.label("entity_uuid"),
+            sa.func.min(table.c.created_on).label("survivor_created_on"),
+        )
+        .select_from(
+            table.outerjoin(
+                boundary,
+                sa.and_(
+                    table.c.entity_type == boundary.c.entity_type,
+                    table.c.entity_uuid == boundary.c.entity_uuid,
+                ),
+            )
+        )
+        .where(table.c.status == STATUS_BLOCKED)
+        .where(table.c.entity_uuid.is_not(None))
+        .where(table.c.created_on <= now)
+        .where(
+            sa.or_(
+                boundary.c.boundary.is_(None),
+                table.c.created_on > boundary.c.boundary,
+            )
+        )
+        .group_by(table.c.entity_type, table.c.entity_uuid)
+        .subquery("streak_survivor")
+    )
+
+
+def _preceded_by_unresolved_attempt(table: sa.Table) -> sa.ColumnElement[bool]:
+    """Whether an unresolved (``pending``) attempt precedes this row.
+
+    A ``pending`` row is the only thing that can insert a streak boundary
+    into *history*: every other write lands at ``now``, newer than every
+    existing row, whereas reconciliation and the purge path finalize a
+    pending row **in place**, keeping its original ``created_on``. So a
+    pending row sitting between two blocked rows is a boundary that may
+    appear at any moment, and the blocked row after it would become the new
+    streak's survivor — the very row pruning must never delete.
+    """
+    pending: sa.Table = table.alias("unresolved_attempt")
+    return sa.exists(
+        sa.select(sa.literal(1))
+        .select_from(pending)
+        .where(
+            sa.and_(
+                pending.c.status == STATUS_PENDING,
+                pending.c.entity_type == table.c.entity_type,
+                pending.c.entity_uuid == table.c.entity_uuid,
+                pending.c.created_on < table.c.created_on,

Review Comment:
   @mikebridge was this tackled in your subsequent commits? If so, mark as resolved, but if you still intend to tackle it, say the word. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org


Re: [PR] feat(deletion-retention): scheduled pruning policy for the purge audit log [superset]

Posted by "bito-code-review[bot] (via GitHub)" <gi...@apache.org>.
bito-code-review[bot] commented on PR #43490:
URL: https://github.com/apache/superset/pull/43490#issuecomment-5535167815

   AI Code Review is in progress (usually takes 3 to 15 minutes unless it's a very large PR).
   
   <details>
   <summary>
   Bito Usage Guide
   </summary>
   
   **Commands**
         
   Type the following command in the pull request comment and save the comment.
         
   - `/review` - Manually triggers an incremental AI Review.
   - `/review full` - Manually triggers a full AI Review.
   
   - `/pause` - Pauses automatic reviews on this pull request.
   - `/resume` - Resumes automatic reviews.
   - `/resolve` - Marks all Bito-posted review comments as resolved.
   - `/abort` - Cancels all in-progress reviews.
   
         
   Refer to the <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/available-commands" target="_blank" rel="noopener noreferrer">documentation</a> for additional commands.
         
   **Configuration**
   
   This repository uses `Superset` You can customize the agent settings <a href="https://alpha.bito.ai/home/ai-agents/code-review-agent" target="_blank" rel="noopener noreferrer">here</a> or contact your Bito workspace admin at evan@preset.io.
         
   **Documentation & Help**
   - <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/getting-started/install-run-using-bito-cloud/create-or-customize-an-agent-instance" target="_blank" rel="noopener noreferrer">Customize agent settings</a>
   - <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/implementing-custom-code-review-rules" target="_blank" rel="noopener noreferrer">Review rules</a>
   - <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent" target="_blank" rel="noopener noreferrer">General documentation</a>
   - <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/faqs" target="_blank" rel="noopener noreferrer">FAQ</a>
   
   </details>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org


Re: [PR] feat(deletion-retention): scheduled pruning policy for the purge audit log [superset]

Posted by "bito-code-review[bot] (via GitHub)" <gi...@apache.org>.
bito-code-review[bot] commented on PR #43490:
URL: https://github.com/apache/superset/pull/43490#issuecomment-5403856238

   AI Code Review is in progress (usually takes 3 to 15 minutes unless it's a very large PR).
   
   <details>
   <summary>
   Bito Usage Guide
   </summary>
   
   **Commands**
         
   Type the following command in the pull request comment and save the comment.
         
   - `/review` - Manually triggers an incremental AI Review.
   - `/review full` - Manually triggers a full AI Review.
   
   - `/pause` - Pauses automatic reviews on this pull request.
   - `/resume` - Resumes automatic reviews.
   - `/resolve` - Marks all Bito-posted review comments as resolved.
   - `/abort` - Cancels all in-progress reviews.
   
         
   Refer to the <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/available-commands" target="_blank" rel="noopener noreferrer">documentation</a> for additional commands.
         
   **Configuration**
   
   This repository uses `Superset` You can customize the agent settings <a href="https://alpha.bito.ai/home/ai-agents/code-review-agent" target="_blank" rel="noopener noreferrer">here</a> or contact your Bito workspace admin at evan@preset.io.
         
   **Documentation & Help**
   - <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/getting-started/install-run-using-bito-cloud/create-or-customize-an-agent-instance" target="_blank" rel="noopener noreferrer">Customize agent settings</a>
   - <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/implementing-custom-code-review-rules" target="_blank" rel="noopener noreferrer">Review rules</a>
   - <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent" target="_blank" rel="noopener noreferrer">General documentation</a>
   - <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/faqs" target="_blank" rel="noopener noreferrer">FAQ</a>
   
   </details>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org


Re: [PR] feat(deletion-retention): scheduled pruning policy for the purge audit log [superset]

Posted by "netlify[bot] (via GitHub)" <gi...@apache.org>.
netlify[bot] commented on PR #43490:
URL: https://github.com/apache/superset/pull/43490#issuecomment-5401523888

   ### <span aria-hidden="true">✅</span> Deploy Preview for *superset-docs-preview* ready!
   
   
   |  Name | Link |
   |:-:|------------------------|
   |<span aria-hidden="true">🔨</span> Latest commit | e2e624ffac7876027eb3a9d4b6ba342a08ed6031 |
   |<span aria-hidden="true">🔍</span> Latest deploy log | https://app.netlify.com/projects/superset-docs-preview/deploys/6a8cb528fd7c3a000824c7db |
   |<span aria-hidden="true">😎</span> Deploy Preview | [https://deploy-preview-43490--superset-docs-preview.netlify.app](https://deploy-preview-43490--superset-docs-preview.netlify.app) |
   |<span aria-hidden="true">📱</span> Preview on mobile | <details><summary> Toggle QR Code... </summary><br /><br />![QR Code](https://app.netlify.com/qr-code/eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJ1cmwiOiJodHRwczovL2RlcGxveS1wcmV2aWV3LTQzNDkwLS1zdXBlcnNldC1kb2NzLXByZXZpZXcubmV0bGlmeS5hcHAifQ.j5aDCv0a3PTmlWW_99i6LVcbzSSkejC5aFxeJjh1jt4)<br /><br />_Use your smartphone camera to open QR code link._</details> |
   |<span aria-hidden="true">🤖</span> Make changes | [Run an agent on this branch](https://app.netlify.com/projects/superset-docs-preview/agent-runs#DcpJDoAgDADAv_QsCbgA8Qc-o7S4JIoNysEY_y63OcwLks9DbhgBGggZE63VFyljrNNGSclLVFh4u5Xkkra01MhR9vOZuFaLnsLQ-pkddai19m1PjgN8Pw) |
   ---
   <!-- [superset-docs-preview Preview](https://deploy-preview-43490--superset-docs-preview.netlify.app) -->
   _To edit notification comments on pull requests, go to your [Netlify project configuration](https://app.netlify.com/projects/superset-docs-preview/configuration/notifications#deploy-notifications)._


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org


Re: [PR] feat(deletion-retention): scheduled pruning policy for the purge audit log [superset]

Posted by "mikebridge (via GitHub)" <gi...@apache.org>.
mikebridge commented on code in PR #43490:
URL: https://github.com/apache/superset/pull/43490#discussion_r3959815436


##########
UPDATING.md:
##########
@@ -63,6 +63,41 @@ payload. Clients must display the new impact and obtain renewed confirmation
 before retrying. Preview or recheck failures fail closed rather than treating
 unknown impact as zero. Chart and dashboard purge endpoints are unchanged.
 
+- The purge audit log can now be pruned automatically. The new

Review Comment:
   Fixed in `5706b963db`. You're right — the operator guide at line ~942 still asserted the `purge_audit_log` is "never pruned by design" and must be pruned manually, contradicting the automatic pruning this PR adds. I reworked that paragraph to describe the new policy accurately: automatic pruning via the `deletion_retention.prune_purge_audit` beat task, the earliest "blocked since" record in a live streak and completed-destruction evidence (`confirmed`/`target_absent`) preserved, only operational noise (`blocked` from resolved streaks + `failed`) aged out past `PURGE_AUDIT_RETENTION_DAYS`, and `PURGE_AUDIT_PRUNING_ENABLED = False` to restore the previous never-pruned behavior. The `## Next` release-note entry (line ~205) and the operator section now agree.
   
   Also in this push: rebased onto master and re-chained the pruning migration onto the current tip `7e2c9a4f1b83` (a sibling migration had merged onto the shared parent and created a second alembic head; `alembic-single-head` is green again).
   
   When you have a moment, could you re-review? Your three over-deletion findings are all addressed (the force-purge duplicate-prune fix in `0cba9314a7`/`d1c1f5e6c9`, plus the two proven-covered cases), and I'd rather this land on your approval than the stale one — the visible green is rusackas' approval from before those fixes.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org


Re: [PR] feat(deletion-retention): scheduled pruning policy for the purge audit log [superset]

Posted by "mikebridge (via GitHub)" <gi...@apache.org>.
mikebridge commented on code in PR #43490:
URL: https://github.com/apache/superset/pull/43490#discussion_r3970511405


##########
superset/commands/deletion_retention/prune_audit.py:
##########
@@ -0,0 +1,689 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Deletion-only pruning of the ``purge_audit_log`` table.
+
+Bounds the audit history's growth without ever weakening its evidentiary
+value. Three delete categories, applied in priority order under one shared
+per-run batch budget:
+
+1. **Blocked duplicates** — within an entity's *current* blockage streak
+   (its ``blocked`` rows newer than the entity's newest streak-breaking
+   row), only the first row of each run of consecutive same-reason rows
+   survives: the streak's earliest row and the first row after every
+   change of block reason. Later same-reason repeats are removed
+   regardless of age. The survivors carry the "blocked since" fact and
+   the reason history — for coded reasons, the same rows the audit writer's
+   own suppression rule retains (reason-less legacy runs are additionally
+   collapsed to their earliest here) — and are never deleted while the
+   streak is current.
+2. **Operational expiry** — ``blocked`` rows of *resolved* streaks and
+   ``failed`` rows older than ``PURGE_AUDIT_OPERATIONAL_RETENTION_DAYS``
+   age out.
+3. **Evidence expiry** — ``confirmed`` / ``target_absent`` rows are
+   untouchable unless ``PURGE_AUDIT_EVIDENCE_RETENTION_DAYS`` is
+   explicitly set (the operator's compliance assertion), and then only
+   rows older than that window.
+
+``pending`` rows belong to :func:`audit.reconcile_pending` and rows with
+future timestamps (clock skew) are excluded from every category *and* from
+streak classification, so a skewed writer cannot reclassify a live streak.
+
+Three invariants keep the survivor safe without a distributed lock, which
+matters because runs can overlap and both ``reconcile_pending`` and the
+purge path finalize rows concurrently. A duplicate is only ever deleted
+when no concurrent transition could turn it into a survivor first:
+
+* **A boundary is never removed while it still bounds anything.** Evidence
+  expiry refuses to delete a row while an older ``blocked`` or ``pending``
+  row for the same entity survives. Boundaries therefore never *recede*,
+  which would otherwise promote resolved rows into a current streak.
+* **A blocked row whose classification is unstable is never deleted.**
+  Finalizing a ``pending`` row resolves it *in place*, keeping its original
+  timestamp, so an unresolved attempt is a boundary that may appear
+  mid-history at any moment — and the blocked row after it would become its
+  streak's survivor. Blocked rows preceded by an unresolved attempt are
+  therefore skipped by **both** the duplicate and the operational category.
+  Age does not stabilize the classification: an old blocked row inside a
+  live streak is exactly the "blocked for years" case FR-009 protects.
+  Without this, a boundary moving *forward* would demote the current
+  survivor and promote the next row into its place — possibly a row already
+  selected for deletion.
+* **Deletes are conditional and counted from rowcounts.** A row whose
+  status changed since selection is not matched, so overlapping runs can
+  neither double-remove nor double-report.
+
+Boundaries moving forward past *all* of an entity's blocked rows leave no
+survivor to promote, so a selected duplicate may be removed slightly ahead
+of its retention window in that case. It was redundant either way and the
+streak's earliest row is untouched.
+
+Timestamps order rows. Ties are possible — legacy second-precision rows,
+two writers within one clock tick — and are resolved on the preserving
+side: a ``pending`` row tied with a blocked row counts as preceding it (the
+block is deferred until the attempt resolves); a blocked row tied with a
+boundary sits on the boundary's *resolved* side (it ages out instead of
+seeding a new current streak, and the boundary is not removed before it);
+and tied same-reason blocked rows are all retained.
+
+Candidate selection is embedded in each ``DELETE`` statement. The derived-table
+wrapper keeps that shape legal on MySQL while ensuring that a pending or
+recovered row committed before the delete is evaluated participates in the
+survivor and boundary predicates. No stale list of candidate ids crosses a
+transaction boundary.
+
+Audit creation/recovery and every pruning batch take the same singleton
+database write lock before assigning a timestamp or evaluating candidates.
+The lock is held through commit, so an audit row cannot become visible in an
+already-processed logical past. Automatic pruning still ships disabled by
+default so operators explicitly choose their retention policy.
+"""
+
+from __future__ import annotations
+
+import logging
+from collections.abc import Callable
+from dataclasses import dataclass, field
+from datetime import datetime, timedelta
+from functools import partial
+from typing import Any, Literal, NamedTuple, TypeAlias
+
+import sqlalchemy as sa
+from flask import current_app
+
+from superset import db
+from superset.commands.deletion_retention.audit import (
+    acquire_coordination_lock,
+    utc_now,
+)
+from superset.models.purge_audit_log import (
+    PurgeAuditLog,
+    STATUS_BLOCKED,
+    STATUS_CONFIRMED,
+    STATUS_FAILED,
+    STATUS_PENDING,
+    STATUS_TARGET_ABSENT,
+)
+
+logger: logging.Logger = logging.getLogger(__name__)
+
+#: Operational records: noise-prone outcomes whose compliance value decays —
+#: a blocked or failed purge leaves the object in place (FR-001).
+OPERATIONAL_STATUSES: frozenset[str] = frozenset({STATUS_BLOCKED, STATUS_FAILED})
+#: Protected evidence: the only surviving trace of a destroyed object.
+PROTECTED_STATUSES: frozenset[str] = frozenset({STATUS_CONFIRMED, STATUS_TARGET_ABSENT})
+#: The outcomes that end a blockage streak — proof the object is gone.
+#:
+#: ``failed`` is deliberately absent. A failed purge is an infrastructure
+#: outcome (the cascade raised), not evidence the blockage cleared: the
+#: policy that blocked the entity is untouched, so the blockage continues
+#: across it. Treating ``failed`` as a boundary would let one transient
+#: error demote the "blocked since" survivor to an ageing duplicate and
+#: restate the blockage as beginning after the failure — losing exactly the
+#: fact FR-003 exists to preserve. ``pending`` is provisional and likewise
+#: neither joins nor breaks streaks.
+_STREAK_BREAKING_STATUSES: frozenset[str] = frozenset(
+    {STATUS_CONFIRMED, STATUS_TARGET_ABSENT}
+)
+
+#: Rows deleted per statement, matching the purge task's batch convention.
+BATCH_SIZE: int = 500
+#: One shared budget for the whole run across all three categories; the
+#: remaining backlog carries over to the next scheduled run (FR-004/SC-004).
+MAX_BATCHES_PER_RUN: int = 10
+
+OPERATIONAL_RETENTION_KEY: str = "PURGE_AUDIT_OPERATIONAL_RETENTION_DAYS"
+EVIDENCE_RETENTION_KEY: str = "PURGE_AUDIT_EVIDENCE_RETENTION_DAYS"
+
+
+class ResolvedWindow(NamedTuple):
+    """A retention window resolved from config.
+
+    Distinguishes the three outcomes a caller must tell apart: a usable
+    window, the deliberate "off" default, and operator error. Collapsing
+    the last two into a bare ``None`` would force every caller to re-read
+    config to find out which it got.
+    """
+
+    days: int | None
+    invalid_key: str | None = None
+
+
+def _validated_window(key: str, value: Any) -> ResolvedWindow:
+    """Validate a configured day count, failing closed on anything odd.
+
+    An invalid value disables its category for the run rather than widening
+    removal (FR-005/SC-005).
+    """
+    # bool is an int subclass and floats would silently truncate — both are
+    # config mistakes, not day counts, on a knob that deletes rows.
+    if not isinstance(value, bool) and not isinstance(value, float):
+        try:
+            days: int = int(value)
+        except (TypeError, ValueError):
+            days = 0
+        if days > 0:
+            return ResolvedWindow(days)
+    logger.warning(
+        "prune_audit: invalid %s=%r; skipping this category for the run "
+        "(pruning never widens on bad configuration)",
+        key,
+        value,
+    )
+    return ResolvedWindow(None, key)
+
+
+def resolve_operational_retention_days() -> ResolvedWindow:
+    """The operational retention window, or a disabled window when invalid."""
+    return _validated_window(
+        OPERATIONAL_RETENTION_KEY, current_app.config.get(OPERATIONAL_RETENTION_KEY)
+    )
+
+
+def resolve_evidence_retention_days() -> ResolvedWindow:
+    """The evidence expiration window; disabled unless explicitly opted in.
+
+    Unset is the documented "never expire evidence" default (FR-006), not an
+    error, so it produces a disabled window with no warning.
+    """
+    value: Any = current_app.config.get(EVIDENCE_RETENTION_KEY)
+    if value is None:
+        return ResolvedWindow(None)
+    return _validated_window(EVIDENCE_RETENTION_KEY, value)
+
+
+@dataclass
+class PruneRunResult:
+    """Per-category removal counts and run disposition for one pruning run."""
+
+    blocked_duplicates: int = 0
+    operational_expired: int = 0
+    evidence_expired: int = 0
+    #: True when the shared batch budget ran out before every category's
+    #: candidates were drained; the remainder converges on later runs.
+    carried_over: bool = False
+    invalid_config_keys: list[str] = field(default_factory=list)
+
+    @property
+    def total_removed(self) -> int:
+        """Total rows removed across every category."""
+        return (
+            self.blocked_duplicates + self.operational_expired + self.evidence_expired
+        )
+
+    def as_dict(self) -> dict[str, Any]:
+        """The task-return / log-line shape of this result."""
+        return {
+            "removed": {
+                "blocked_duplicates": self.blocked_duplicates,
+                "operational_expired": self.operational_expired,
+                "evidence_expired": self.evidence_expired,
+            },
+            "carried_over": self.carried_over,
+            "invalid_config_keys": list(self.invalid_config_keys),
+        }
+
+
+def _streak_boundary_subquery(now: datetime) -> sa.Subquery:
+    """Per entity, the ``created_on`` of its newest streak-breaking row.
+
+    Entities absent from this subquery have never been proven destroyed —
+    all their blocked rows form one current streak. Future-dated rows are
+    excluded so a skewed writer clock cannot push the boundary ahead of a
+    live streak and make its rows look resolved.
+    """
+    table: sa.Table = PurgeAuditLog.__table__
+    return (
+        sa.select(
+            table.c.entity_type.label("entity_type"),
+            table.c.entity_uuid.label("entity_uuid"),
+            sa.func.max(table.c.created_on).label("boundary"),
+        )
+        .where(table.c.status.in_(_STREAK_BREAKING_STATUSES))
+        .where(table.c.entity_uuid.is_not(None))
+        .where(table.c.created_on <= now)
+        .group_by(table.c.entity_type, table.c.entity_uuid)
+        .subquery("streak_boundary")
+    )
+
+
+def _with_boundary(table: sa.Table, boundary: sa.Subquery) -> sa.Join:
+    """Outer-join each row to its entity's streak boundary (NULL if none)."""
+    return table.outerjoin(
+        boundary,
+        sa.and_(
+            table.c.entity_type == boundary.c.entity_type,
+            table.c.entity_uuid == boundary.c.entity_uuid,
+        ),
+    )
+
+
+def _in_current_streak(
+    row: sa.FromClause, boundary: sa.Subquery
+) -> sa.ColumnElement[bool]:
+    """Whether ``row`` is newer than its entity's boundary (or there is none).
+
+    Strictly newer: a row tied with the boundary sits on its resolved side.
+    A boundary proves the object was gone at that instant, and a block that
+    cannot be ordered after the destruction must not seed a new "current"
+    streak — that would mint a survivor exempt from age-out forever for an
+    object that no longer exists.
+    """
+    return sa.or_(boundary.c.boundary.is_(None), row.c.created_on > boundary.c.boundary)
+
+
+def _repeats_an_earlier_block(
+    table: sa.Table, boundary: sa.Subquery
+) -> sa.ColumnElement[bool]:
+    """Whether a same-reason current-streak block precedes this row with no
+    change of reason in between.
+
+    This is the audit writer's suppression rule
+    (:func:`audit.finalize_retention_blocked`) applied retroactively: a
+    block repeating the reason of the block just before it adds nothing,
+    while the first block after a reason change is the only durable record
+    of the new cause and is retained. The comparison is NULL-safe, so
+    consecutive reason-less (pre-feature) rows count as one run, deduped to
+    the run's earliest, and the first coded block ends that run. Note this
+    is *stricter* than the writer for reason-less rows: the writer never
+    suppresses a reason-less block (``_suppress_redundant_block`` bails on a
+    missing code), so here the pruner additionally collapses legacy
+    pre-feature duplicates — the streak's earliest "blocked since" row is
+    still always kept. Tied same-reason rows are not "earlier" than each
+    other, so all of them are kept.
+    """
+    earlier: sa.FromClause = table.alias("earlier_block")
+    between: sa.FromClause = table.alias("reason_change")
+    reason_changed_between: sa.ColumnElement[bool] = sa.exists(
+        sa.select(sa.literal(1))
+        .select_from(between)
+        .where(
+            sa.and_(
+                between.c.status == STATUS_BLOCKED,
+                between.c.entity_type == table.c.entity_type,
+                between.c.entity_uuid == table.c.entity_uuid,
+                # Inclusive bounds: a differing-reason block sharing an
+                # exact timestamp with either endpoint still breaks the run,
+                # so a reason-transition row tied with a neighbour is
+                # preserved as a run head rather than pruned as a repeat
+                # (the same preserving-side tie rule the pending and evidence
+                # guards use). Inclusive bounds only ever add boundaries —
+                # i.e. only ever preserve more, never delete more.
+                between.c.created_on >= earlier.c.created_on,
+                between.c.created_on <= table.c.created_on,
+                between.c.reason.is_distinct_from(table.c.reason),
+            )
+        )
+        .correlate(table, earlier)
+    )
+    return sa.exists(
+        sa.select(sa.literal(1))
+        .select_from(earlier)
+        .where(
+            sa.and_(
+                earlier.c.status == STATUS_BLOCKED,
+                earlier.c.entity_type == table.c.entity_type,
+                earlier.c.entity_uuid == table.c.entity_uuid,
+                _in_current_streak(earlier, boundary),
+                earlier.c.created_on < table.c.created_on,
+                earlier.c.reason.is_not_distinct_from(table.c.reason),
+                sa.not_(reason_changed_between),
+            )
+        )
+        .correlate(table, boundary)
+    )
+
+
+def _preceded_by_unresolved_attempt(table: sa.Table) -> sa.ColumnElement[bool]:
+    """Whether an unresolved (``pending``) attempt precedes this row.
+
+    A ``pending`` row is the only thing that can insert a streak boundary
+    into *history*: every other write lands at ``now``, newer than every
+    existing row, whereas reconciliation and the purge path finalize a
+    pending row **in place**, keeping its original ``created_on``. So a
+    pending row sitting between two blocked rows is a boundary that may
+    appear at any moment, and the blocked row after it would become the new
+    streak's survivor — the very row pruning must never delete.
+
+    A tied timestamp counts as preceding. Which of the two writes landed
+    first is unknowable from the row, the block's classification (current
+    duplicate, or resolved-streak row on an age window) changes with the
+    attempt's outcome, and deferring it until then costs nothing.
+    """
+    pending: sa.FromClause = table.alias("unresolved_attempt")
+    return sa.exists(
+        sa.select(sa.literal(1))
+        .select_from(pending)
+        .where(
+            sa.and_(
+                pending.c.status == STATUS_PENDING,
+                pending.c.entity_type == table.c.entity_type,
+                pending.c.entity_uuid == table.c.entity_uuid,
+                pending.c.created_on <= table.c.created_on,
+            )
+        )
+    )
+
+
+def _duplicate_candidates(now: datetime, limit: int) -> sa.sql.Select:
+    """Select current-streak blocked rows that repeat the block before them.
+
+    Age-independent by design (FR-003): a repeat is prunable the moment the
+    streak holds an earlier same-reason block, regardless of the retention
+    window. Trigger is not a discriminator — ``scheduled`` and ``force``
+    blocked rows share streaks. Reason *is*: the first block after a reason
+    change survives alongside the streak's earliest row (see
+    :func:`_repeats_an_earlier_block`).
+
+    Rows preceded by an unresolved attempt are skipped: their classification
+    is not stable, because that attempt can finalize into a boundary and
+    promote them to survivor between selection and deletion. Such rows are
+    collected once the attempt resolves. Note that pruning does not depend on
+    that happening promptly — reconciliation runs from the purge task, which
+    a deployment may have disabled or left in dry-run — so a long-lived
+    pending row defers its successors indefinitely rather than risking them.
+    """
+    table: sa.Table = PurgeAuditLog.__table__
+    boundary: sa.Subquery = _streak_boundary_subquery(now)
+    return (
+        sa.select(table.c.id)
+        .select_from(_with_boundary(table, boundary))
+        .where(table.c.status == STATUS_BLOCKED)

Review Comment:
   Done in `bcfa5055e1` — went with full immortality per your call. `_operational_candidates` now excludes `status == BLOCKED AND trigger == FORCE`, so a force block is never pruned by either category (duplicate collapse *or* operational age-out) — it survives permanently, not just while its streak is current. `failed` rows (force-triggered or not) still age normally. Updated the `_repeats_an_earlier_block` / `_operational_candidates` docstrings that used to say a resolved-streak force block ages out, and inverted the old characterization test into `test_force_block_survives_operational_window_permanently` (a 100-day-old resolved-streak force block survives a 90-day operational prune). The longer-but-not-forever cleanup pass is left as a possible follow-up, out of scope here.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org


Re: [PR] feat(deletion-retention): scheduled pruning policy for the purge audit log [superset]

Posted by "bito-code-review[bot] (via GitHub)" <gi...@apache.org>.
bito-code-review[bot] commented on PR #43490:
URL: https://github.com/apache/superset/pull/43490#issuecomment-5640304730

   AI Code Review is in progress (usually takes 3 to 15 minutes unless it's a very large PR).
   
   <details>
   <summary>
   Bito Usage Guide
   </summary>
   
   **Commands**
         
   Type the following command in the pull request comment and save the comment.
         
   - `/review` - Manually triggers an incremental AI Review.
   - `/review full` - Manually triggers a full AI Review.
   
   - `/pause` - Pauses automatic reviews on this pull request.
   - `/resume` - Resumes automatic reviews.
   - `/resolve` - Marks all Bito-posted review comments as resolved.
   - `/abort` - Cancels all in-progress reviews.
   
         
   Refer to the <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/available-commands" target="_blank" rel="noopener noreferrer">documentation</a> for additional commands.
         
   **Configuration**
   
   This repository uses `Superset` You can customize the agent settings <a href="https://alpha.bito.ai/home/ai-agents/code-review-agent" target="_blank" rel="noopener noreferrer">here</a> or contact your Bito workspace admin at evan@preset.io.
         
   **Documentation & Help**
   - <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/getting-started/install-run-using-bito-cloud/create-or-customize-an-agent-instance" target="_blank" rel="noopener noreferrer">Customize agent settings</a>
   - <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/implementing-custom-code-review-rules" target="_blank" rel="noopener noreferrer">Review rules</a>
   - <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent" target="_blank" rel="noopener noreferrer">General documentation</a>
   - <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/faqs" target="_blank" rel="noopener noreferrer">FAQ</a>
   
   </details>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org


Re: [PR] feat(deletion-retention): scheduled pruning policy for the purge audit log [superset]

Posted by "mikebridge (via GitHub)" <gi...@apache.org>.
mikebridge commented on code in PR #43490:
URL: https://github.com/apache/superset/pull/43490#discussion_r3979162577


##########
superset/commands/deletion_retention/prune_audit.py:
##########
@@ -0,0 +1,726 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Deletion-only pruning of the ``purge_audit_log`` table.
+
+Bounds the audit history's growth without ever weakening its evidentiary
+value. Three delete categories, applied in priority order under one shared
+per-run batch budget:
+
+1. **Blocked duplicates** — within an entity's *current* blockage streak
+   (its ``blocked`` rows newer than the entity's newest streak-breaking
+   row), only the first row of each run of consecutive same-reason rows
+   survives: the streak's earliest row and the first row after every
+   change of block reason. Later same-reason repeats are removed
+   regardless of age. The survivors carry the "blocked since" fact and
+   the reason history — for coded reasons, the same rows the audit writer's
+   own suppression rule retains (reason-less legacy runs are additionally
+   collapsed to their earliest here) — and are never deleted while the
+   streak is current.
+2. **Operational expiry** — ``blocked`` rows of *resolved* streaks and
+   ``failed`` rows older than ``PURGE_AUDIT_OPERATIONAL_RETENTION_DAYS``
+   age out.
+3. **Evidence expiry** — ``confirmed`` / ``target_absent`` rows are
+   untouchable unless ``PURGE_AUDIT_EVIDENCE_RETENTION_DAYS`` is
+   explicitly set (the operator's compliance assertion), and then only
+   rows older than that window.
+
+``pending`` rows belong to :func:`audit.reconcile_pending` and rows with
+future timestamps (clock skew) are excluded from every category *and* from
+streak classification, so a skewed writer cannot reclassify a live streak.
+
+Three invariants keep the survivor safe without a distributed lock, which
+matters because runs can overlap and both ``reconcile_pending`` and the
+purge path finalize rows concurrently. A duplicate is only ever deleted
+when no concurrent transition could turn it into a survivor first:
+
+* **A boundary is never removed while it still bounds anything.** Evidence
+  expiry refuses to delete a row while an older ``blocked`` or ``pending``
+  row for the same entity survives. Boundaries therefore never *recede*,
+  which would otherwise promote resolved rows into a current streak.
+* **A blocked row whose classification is unstable is never deleted.**
+  Finalizing a ``pending`` row resolves it *in place*, keeping its original
+  timestamp, so an unresolved attempt is a boundary that may appear
+  mid-history at any moment — and the blocked row after it would become its
+  streak's survivor. Blocked rows preceded by an unresolved attempt are
+  therefore skipped by **both** the duplicate and the operational category.
+  Age does not stabilize the classification: an old blocked row inside a
+  live streak is exactly the "blocked for years" case FR-009 protects.
+  Without this, a boundary moving *forward* would demote the current
+  survivor and promote the next row into its place — possibly a row already
+  selected for deletion.
+* **Deletes are conditional and counted from rowcounts.** A row whose
+  status changed since selection is not matched, so overlapping runs can
+  neither double-remove nor double-report.
+
+Boundaries moving forward past *all* of an entity's blocked rows leave no
+survivor to promote, so a selected duplicate may be removed slightly ahead
+of its retention window in that case. It was redundant either way and the
+streak's earliest row is untouched.
+
+Timestamps order rows. Ties are possible — legacy second-precision rows,
+two writers within one clock tick — and are resolved on the preserving
+side: a ``pending`` row tied with a blocked row counts as preceding it (the
+block is deferred until the attempt resolves); a blocked row tied with a
+boundary sits on the boundary's *resolved* side (it ages out instead of
+seeding a new current streak, and the boundary is not removed before it);
+and tied same-reason blocked rows are all retained.
+
+Candidate selection is embedded in each ``DELETE`` statement. The derived-table
+wrapper keeps that shape legal on MySQL while ensuring that a pending or
+recovered row committed before the delete is evaluated participates in the
+survivor and boundary predicates. No stale list of candidate ids crosses a
+transaction boundary.
+
+Audit creation/recovery and every pruning batch take the same singleton
+database write lock before assigning a timestamp or evaluating candidates.
+The lock is held through commit, so an audit row cannot become visible in an
+already-processed logical past. Automatic pruning still ships disabled by
+default so operators explicitly choose their retention policy.
+"""
+
+from __future__ import annotations
+
+import logging
+from collections.abc import Callable
+from dataclasses import dataclass, field
+from datetime import datetime, timedelta
+from functools import partial
+from typing import Any, Literal, NamedTuple, TypeAlias
+
+import sqlalchemy as sa
+from flask import current_app
+
+from superset import db
+from superset.commands.deletion_retention.audit import (
+    acquire_coordination_lock,
+    TRIGGER_FORCE,
+    utc_now,
+)
+from superset.models.purge_audit_log import (
+    PurgeAuditLog,
+    STATUS_BLOCKED,
+    STATUS_CONFIRMED,
+    STATUS_FAILED,
+    STATUS_PENDING,
+    STATUS_TARGET_ABSENT,
+)
+
+logger: logging.Logger = logging.getLogger(__name__)
+
+#: Operational records: noise-prone outcomes whose compliance value decays —
+#: a blocked or failed purge leaves the object in place (FR-001).
+OPERATIONAL_STATUSES: frozenset[str] = frozenset({STATUS_BLOCKED, STATUS_FAILED})
+#: Protected evidence: the only surviving trace of a destroyed object.
+PROTECTED_STATUSES: frozenset[str] = frozenset({STATUS_CONFIRMED, STATUS_TARGET_ABSENT})
+#: The outcomes that end a blockage streak — proof the object is gone.
+#:
+#: ``failed`` is deliberately absent. A failed purge is an infrastructure
+#: outcome (the cascade raised), not evidence the blockage cleared: the
+#: policy that blocked the entity is untouched, so the blockage continues
+#: across it. Treating ``failed`` as a boundary would let one transient
+#: error demote the "blocked since" survivor to an ageing duplicate and
+#: restate the blockage as beginning after the failure — losing exactly the
+#: fact FR-003 exists to preserve. ``pending`` is provisional and likewise
+#: neither joins nor breaks streaks.
+_STREAK_BREAKING_STATUSES: frozenset[str] = frozenset(
+    {STATUS_CONFIRMED, STATUS_TARGET_ABSENT}
+)
+
+#: Rows deleted per statement, matching the purge task's batch convention.
+BATCH_SIZE: int = 500
+#: One shared budget for the whole run across all three categories; the
+#: remaining backlog carries over to the next scheduled run (FR-004/SC-004).
+MAX_BATCHES_PER_RUN: int = 10
+
+OPERATIONAL_RETENTION_KEY: str = "PURGE_AUDIT_OPERATIONAL_RETENTION_DAYS"
+EVIDENCE_RETENTION_KEY: str = "PURGE_AUDIT_EVIDENCE_RETENTION_DAYS"
+
+
+class ResolvedWindow(NamedTuple):
+    """A retention window resolved from config.
+
+    Distinguishes the three outcomes a caller must tell apart: a usable
+    window, the deliberate "off" default, and operator error. Collapsing
+    the last two into a bare ``None`` would force every caller to re-read
+    config to find out which it got.
+    """
+
+    days: int | None
+    invalid_key: str | None = None
+
+
+def _validated_window(key: str, value: Any) -> ResolvedWindow:
+    """Validate a configured day count, failing closed on anything odd.
+
+    An invalid value disables its category for the run rather than widening
+    removal (FR-005/SC-005).
+    """
+    # bool is an int subclass and floats would silently truncate — both are
+    # config mistakes, not day counts, on a knob that deletes rows.
+    if not isinstance(value, bool) and not isinstance(value, float):
+        try:
+            days: int = int(value)
+        except (TypeError, ValueError):
+            days = 0
+        if days > 0:
+            return ResolvedWindow(days)
+    logger.warning(
+        "prune_audit: invalid %s=%r; skipping this category for the run "
+        "(pruning never widens on bad configuration)",
+        key,
+        value,
+    )
+    return ResolvedWindow(None, key)
+
+
+def resolve_operational_retention_days() -> ResolvedWindow:
+    """The operational retention window, or a disabled window when invalid."""
+    return _validated_window(
+        OPERATIONAL_RETENTION_KEY, current_app.config.get(OPERATIONAL_RETENTION_KEY)
+    )
+
+
+def resolve_evidence_retention_days() -> ResolvedWindow:
+    """The evidence expiration window; disabled unless explicitly opted in.
+
+    Unset is the documented "never expire evidence" default (FR-006), not an
+    error, so it produces a disabled window with no warning.
+    """
+    value: Any = current_app.config.get(EVIDENCE_RETENTION_KEY)
+    if value is None:
+        return ResolvedWindow(None)
+    return _validated_window(EVIDENCE_RETENTION_KEY, value)
+
+
+@dataclass
+class PruneRunResult:
+    """Per-category removal counts and run disposition for one pruning run."""
+
+    blocked_duplicates: int = 0
+    operational_expired: int = 0
+    evidence_expired: int = 0
+    #: True when the shared batch budget ran out before every category's
+    #: candidates were drained; the remainder converges on later runs.
+    carried_over: bool = False
+    invalid_config_keys: list[str] = field(default_factory=list)
+
+    @property
+    def total_removed(self) -> int:
+        """Total rows removed across every category."""
+        return (
+            self.blocked_duplicates + self.operational_expired + self.evidence_expired
+        )
+
+    def as_dict(self) -> dict[str, Any]:
+        """The task-return / log-line shape of this result."""
+        return {
+            "removed": {
+                "blocked_duplicates": self.blocked_duplicates,
+                "operational_expired": self.operational_expired,
+                "evidence_expired": self.evidence_expired,
+            },
+            "carried_over": self.carried_over,
+            "invalid_config_keys": list(self.invalid_config_keys),
+        }
+
+
+def _streak_boundary_subquery(now: datetime) -> sa.Subquery:
+    """Per entity, the ``created_on`` of its newest streak-breaking row.
+
+    Entities absent from this subquery have never been proven destroyed —
+    all their blocked rows form one current streak. Future-dated rows are
+    excluded so a skewed writer clock cannot push the boundary ahead of a
+    live streak and make its rows look resolved.
+    """
+    table: sa.Table = PurgeAuditLog.__table__
+    return (
+        sa.select(
+            table.c.entity_type.label("entity_type"),
+            table.c.entity_uuid.label("entity_uuid"),
+            sa.func.max(table.c.created_on).label("boundary"),
+        )
+        .where(table.c.status.in_(_STREAK_BREAKING_STATUSES))
+        .where(table.c.entity_uuid.is_not(None))
+        .where(table.c.created_on <= now)
+        .group_by(table.c.entity_type, table.c.entity_uuid)
+        .subquery("streak_boundary")
+    )
+
+
+def _with_boundary(table: sa.Table, boundary: sa.Subquery) -> sa.Join:
+    """Outer-join each row to its entity's streak boundary (NULL if none)."""
+    return table.outerjoin(
+        boundary,
+        sa.and_(
+            table.c.entity_type == boundary.c.entity_type,
+            table.c.entity_uuid == boundary.c.entity_uuid,
+        ),
+    )
+
+
+def _in_current_streak(
+    row: sa.FromClause, boundary: sa.Subquery
+) -> sa.ColumnElement[bool]:
+    """Whether ``row`` is newer than its entity's boundary (or there is none).
+
+    Strictly newer: a row tied with the boundary sits on its resolved side.
+    A boundary proves the object was gone at that instant, and a block that
+    cannot be ordered after the destruction must not seed a new "current"
+    streak — that would mint a survivor exempt from age-out forever for an
+    object that no longer exists.
+    """
+    return sa.or_(boundary.c.boundary.is_(None), row.c.created_on > boundary.c.boundary)
+
+
+def _repeats_an_earlier_block(
+    table: sa.Table, boundary: sa.Subquery
+) -> sa.ColumnElement[bool]:
+    """Whether a same-reason current-streak block precedes this row with no
+    change of reason in between.
+
+    This is the audit writer's suppression rule
+    (:func:`audit.finalize_retention_blocked`) applied retroactively: a
+    block repeating the reason of the block just before it adds nothing,
+    while the first block after a reason change is the only durable record
+    of the new cause and is retained. The comparison is NULL-safe, so
+    consecutive reason-less (pre-feature) rows count as one run, deduped to
+    the run's earliest, and the first coded block ends that run. Note this
+    is *stricter* than the writer for reason-less rows: the writer never
+    suppresses a reason-less block (``_suppress_redundant_block`` bails on a
+    missing code), so here the pruner additionally collapses legacy
+    pre-feature duplicates — the streak's earliest "blocked since" row is
+    still always kept. Tied same-reason rows are not "earlier" than each
+    other, so all of them are kept.
+
+    A ``force`` row is exempt: it is never reported as a repeat (see the
+    return), so it is kept out of the duplicate category and marked a
+    survivor while its streak is current. It is also exempt from operational
+    age-out (see :func:`_operational_candidates`), so an operator force-purge
+    block is retained permanently — never pruned by either category.
+    """
+    earlier: sa.FromClause = table.alias("earlier_block")
+    between: sa.FromClause = table.alias("reason_change")
+    reason_changed_between: sa.ColumnElement[bool] = sa.exists(
+        sa.select(sa.literal(1))
+        .select_from(between)
+        .where(
+            sa.and_(
+                between.c.status == STATUS_BLOCKED,
+                between.c.entity_type == table.c.entity_type,
+                between.c.entity_uuid == table.c.entity_uuid,
+                # Inclusive bounds: a differing-reason block sharing an
+                # exact timestamp with either endpoint still breaks the run,
+                # so a reason-transition row tied with a neighbour is
+                # preserved as a run head rather than pruned as a repeat
+                # (the same preserving-side tie rule the pending and evidence
+                # guards use). Inclusive bounds only ever add boundaries —
+                # i.e. only ever preserve more, never delete more.
+                between.c.created_on >= earlier.c.created_on,
+                between.c.created_on <= table.c.created_on,
+                between.c.reason.is_distinct_from(table.c.reason),
+            )
+        )
+        .correlate(table, earlier)
+    )
+    repeats: sa.ColumnElement[bool] = sa.exists(
+        sa.select(sa.literal(1))
+        .select_from(earlier)
+        .where(
+            sa.and_(
+                earlier.c.status == STATUS_BLOCKED,
+                earlier.c.entity_type == table.c.entity_type,
+                earlier.c.entity_uuid == table.c.entity_uuid,
+                _in_current_streak(earlier, boundary),
+                earlier.c.created_on < table.c.created_on,
+                earlier.c.reason.is_not_distinct_from(table.c.reason),
+                sa.not_(reason_changed_between),
+            )
+        )
+        .correlate(table, boundary)
+    )
+    # A ``force`` attempt is an operator action the writer never suppresses:
+    # audit.py's ``_suppress_redundant_block`` only collapses consecutive
+    # scheduled same-reason blocks, so the pruner does not collapse a force row
+    # either — it is never reported as a repeat. This keeps it out of the
+    # duplicate category and (via ``sa.not_`` in the operational category) marks
+    # it a survivor while its streak is current. It is also exempt from
+    # operational age-out (``_operational_candidates`` excludes force blocks), so
+    # a force block is retained permanently — full immortality for operator
+    # force-purge blocks. A force row may still be the *earlier* anchor a later
+    # scheduled repeat collapses into — only the force row itself is protected
+    # from duplicate removal.
+    return sa.and_(table.c.trigger != TRIGGER_FORCE, repeats)
+
+
+def _preceded_by_unresolved_attempt(table: sa.Table) -> sa.ColumnElement[bool]:
+    """Whether an unresolved (``pending``) attempt precedes this row.
+
+    A ``pending`` row is the only thing that can insert a streak boundary
+    into *history*: every other write lands at ``now``, newer than every
+    existing row, whereas reconciliation and the purge path finalize a
+    pending row **in place**, keeping its original ``created_on``. So a
+    pending row sitting between two blocked rows is a boundary that may
+    appear at any moment, and the blocked row after it would become the new
+    streak's survivor — the very row pruning must never delete.
+
+    A tied timestamp counts as preceding. Which of the two writes landed
+    first is unknowable from the row, the block's classification (current
+    duplicate, or resolved-streak row on an age window) changes with the
+    attempt's outcome, and deferring it until then costs nothing.
+    """
+    pending: sa.FromClause = table.alias("unresolved_attempt")
+    return sa.exists(
+        sa.select(sa.literal(1))
+        .select_from(pending)
+        .where(
+            sa.and_(
+                pending.c.status == STATUS_PENDING,
+                pending.c.entity_type == table.c.entity_type,
+                pending.c.entity_uuid == table.c.entity_uuid,
+                pending.c.created_on <= table.c.created_on,
+            )
+        )
+    )
+
+
+def _duplicate_candidates(now: datetime, limit: int) -> sa.sql.Select:
+    """Select current-streak blocked rows that repeat the block before them.
+
+    Age-independent by design (FR-003): a repeat is prunable the moment the
+    streak holds an earlier same-reason block, regardless of the retention
+    window. Trigger does not gate streak membership — ``scheduled`` and
+    ``force`` blocked rows share streaks — but a ``force`` row is itself never
+    collapsed as a repeat (see :func:`_repeats_an_earlier_block`). Reason *is*
+    a discriminator: the first block after a reason change survives alongside
+    the streak's earliest row.
+
+    Rows preceded by an unresolved attempt are skipped: their classification
+    is not stable, because that attempt can finalize into a boundary and
+    promote them to survivor between selection and deletion. Such rows are
+    collected once the attempt resolves. Note that pruning does not depend on
+    that happening promptly — reconciliation runs from the purge task, which
+    a deployment may have disabled or left in dry-run — so a long-lived
+    pending row defers its successors indefinitely rather than risking them.
+    """
+    table: sa.Table = PurgeAuditLog.__table__
+    boundary: sa.Subquery = _streak_boundary_subquery(now)
+    return (
+        sa.select(table.c.id)
+        .select_from(_with_boundary(table, boundary))
+        .where(table.c.status == STATUS_BLOCKED)
+        .where(table.c.entity_uuid.is_not(None))
+        .where(table.c.created_on <= now)
+        .where(_in_current_streak(table, boundary))
+        .where(_repeats_an_earlier_block(table, boundary))
+        .where(sa.not_(_preceded_by_unresolved_attempt(table)))
+        .order_by(table.c.created_on)
+        .limit(limit)

Review Comment:
   Good catch — you're right that `BATCH_SIZE` bounded deletions but not the discovery scan, which ran inside the DELETE under the coordination lock. Fixed in ad34aeee0f by splitting discovery from the locked mutation:
   
   - **Discovery runs unlocked.** The ≤`BATCH_SIZE` candidate ids are selected *before* the lock is taken, so the age-unbounded scan and its boundary/`EXISTS` checks — including the sparse/no-duplicate steady state you describe — no longer happen while holding the singleton lock.
   - **Only a bounded re-check + delete run under the lock.** Under the lock the batch is re-checked with a SELECT that re-applies the identical candidacy predicates over just those ≤`BATCH_SIZE` ids, then the survivors are deleted by literal id. Both locked statements are ≤`BATCH_SIZE`-scoped → per-PK and per-entity index probes, so lock-hold no longer scales with table size.
   - **The boundary query is now bounded too.** The streak boundary was rewritten from a global `GROUP BY entity` aggregate to a correlated per-entity `MAX(created_on)` scalar, shared by discovery and the re-check through one predicate list so the two cannot drift.
   - The re-check is a SELECT rather than a self-referencing DELETE subquery because MySQL rejects the latter (ERROR 1093); the discovery transaction is ended before the lock is taken, so the re-check reads current committed state regardless of isolation level (Superset pins MySQL to READ COMMITTED, so shipped config was already safe — this is isolation-independence hardening).
   
   Verification: the existing behavioral `run_prune` integration tests pass unchanged (34/34 on Postgres), plus a new regression test for the discovery→lock re-check that forces REPEATABLE READ and was controlled red-first (reverting the pre-lock transaction reset makes it fail). Would appreciate your ack on the approach.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org


Re: [PR] feat(deletion-retention): scheduled pruning policy for the purge audit log [superset]

Posted by "mikebridge (via GitHub)" <gi...@apache.org>.
mikebridge commented on code in PR #43490:
URL: https://github.com/apache/superset/pull/43490#discussion_r4007924182


##########
UPDATING.md:
##########
@@ -945,7 +980,7 @@ With the flag on, delete confirmations across the chart/dashboard/dataset list p
 
 This also resolves the limitation noted under *Soft delete and restore for datasets*: a database blocked by soft-deleted datasets can now be freed by purging those datasets (per-entity endpoint, retention task, or `force-purge` CLI) instead of hard-deleting `tables` rows out-of-band.
 
-The `purge_audit_log` table is **never pruned by design** — the audit must survive the entities it names; operators who need to age it out should prune manually.
+Automatic pruning of the `purge_audit_log` table is available but **off by default**: set `PURGE_AUDIT_PRUNING_ENABLED = True` to enable the `deletion_retention.prune_purge_audit` Celery beat task (daily, 03:30) so the table no longer grows unbounded and does not need manual pruning. Left at its default (`PURGE_AUDIT_PRUNING_ENABLED = False`) the table is never pruned and grows indefinitely — enabling it is an explicit operator choice. The policy is written to preserve the audit's meaning rather than trade it away: within an entity's current blockage streak the earliest — "blocked since" — record always survives (only redundant duplicate `blocked` records are collapsed), and completed-destruction evidence (`confirmed`, `target_absent`) is **never** removed unless the separate `PURGE_AUDIT_EVIDENCE_RETENTION_DAYS` opt-in is explicitly set. What ages out is operational noise — `blocked` records from already-resolved streaks and `failed` records — once older than `PURGE_AUDIT
 _OPERATIONAL_RETENTION_DAYS` (default 90). See the release-note entry above for the beat-schedule and `CELERY_CONFIG` details.

Review Comment:
   Addressed in 07ade68f7e — the bounded-growth promise is gone; both the release-note summary and the detailed section now state that force-triggered `blocked` records are retained permanently (exempt from duplicate collapse and operational age-out, including in resolved streaks), so repeated force-purge attempts against a persistently blocked entity still add a record each.



##########
UPDATING.md:
##########
@@ -243,6 +243,41 @@ unknown impact as zero. Chart and dashboard purge endpoints are unchanged.
 
 - The dashboard datasource-based visibility fallback now fails closed: a dashboard whose member charts’ datasources cannot be resolved (deleted datasource rows, missing `datasource_id`, or unsupported datasource types) is no longer accessible to users without explicit editor/viewer rights, and a dashboard composed of semantic-view charts now requires `datasource_access` on (at least one of) its semantic views or their parent semantic layer — previously any authenticated user could open such a dashboard’s shell. Because the fallback now considers every member chart rather than only table-backed ones, a user holding `datasource_access` on any single member datasource — including a semantic view or its parent layer — can open a mixed dashboard that previously denied them. Dashboards with no charts remain accessible, and dashboards with explicit viewers are unaffected. Conversely, holders of `all_datasource_access` now see every published no-viewer dashboard in the dashboard l
 ist — including chart-less ones previously hidden by the inner joins — matching what the object-level gate already allowed them to open.
 - Version restore (`POST /api/v1/{chart,dashboard,dataset}/<uuid>/versions/<version_uuid>/restore`) now refuses an **externally managed** entity (`is_managed_externally = True`) with HTTP 403, enforcing server-side what the docs already promised. Previously the refusal existed only in the browser, so an otherwise-authorized editor could restore such an entity by calling the endpoint directly and have the restore overwritten on the next external sync. Soft-delete recovery is deliberately unaffected — it changes visibility, not content.
+- The purge audit log can now be pruned automatically. The new
+  `deletion_retention.prune_purge_audit` Celery beat task (daily, 03:30, in the
+  default `CeleryConfig.beat_schedule`) removes duplicate `blocked` records
+  within an entity's current blockage streak (the earliest — "blocked since" —
+  record and the first record after each change of block `reason` always
+  survive, mirroring the audit writer's own suppression rule) and ages out
+  operational records (`blocked` from
+  resolved streaks, `failed`) older than
+  `PURGE_AUDIT_OPERATIONAL_RETENTION_DAYS` (default 90). A streak is ended
+  only by proof the object is gone (`confirmed`/`target_absent`); a `failed`
+  attempt does not reset the "blocked since" record. Completed-destruction
+  evidence (`confirmed`, `target_absent`) is **never touched** unless the
+  separate `PURGE_AUDIT_EVIDENCE_RETENTION_DAYS` opt-in is explicitly set,
+  which is the operator's assertion that an approved compliance policy
+  permits expiring destruction evidence. Automatic deletion is disabled by
+  default; set `PURGE_AUDIT_PRUNING_ENABLED = True` after reviewing these

Review Comment:
   Addressed in 07ade68f7e — documented as an explicit two-phase rollout: migrations + the coordinated writer code on every audit-writing process (web/API, CLI, Celery) with old in-flight writes drained BEFORE the flag is enabled, and pruning disabled with running batches finished before rolling any writer back.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org


Re: [PR] feat(deletion-retention): scheduled pruning policy for the purge audit log [superset]

Posted by "bito-code-review[bot] (via GitHub)" <gi...@apache.org>.
bito-code-review[bot] commented on code in PR #43490:
URL: https://github.com/apache/superset/pull/43490#discussion_r3848911174


##########
superset/migrations/versions/2026-08-24_15-50_a6c21e5b4d93_index_purge_audit_pruning.py:
##########
@@ -0,0 +1,46 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Index purge-audit pruning predicates.
+
+Revision ID: a6c21e5b4d93
+Revises: 1072de5ed955
+Create Date: 2026-08-24 15:50:00.000000
+
+"""
+
+from superset.migrations.shared.utils import create_index, drop_index
+
+# revision identifiers, used by Alembic.
+revision: str = "a6c21e5b4d93"
+down_revision: str = "1072de5ed955"
+
+_TABLE_NAME: str = "purge_audit_log"
+_INDEX_NAME: str = "ix_purge_audit_log_pruning"
+
+
+def upgrade() -> None:
+    """Add an index matching recurring pruning access patterns."""
+    create_index(
+        _TABLE_NAME,
+        _INDEX_NAME,
+        ["status", "entity_type", "entity_uuid", "created_on"],
+    )
+
+
+def downgrade() -> None:
+    """Remove the purge-audit pruning index."""
+    drop_index(_TABLE_NAME, _INDEX_NAME)

Review Comment:
   <div>
   
   
   <div id="suggestion">
   <div id="issue"><b>Missing unit test coverage for new migration</b></div>
   <div id="fix">
   
   The migration adds a composite index for pruning queries but lacks unit test coverage. The predecessor migration (`b8d2f4a6c901`) has a dedicated test file (`test_purge_audit_predecessor_index.py`) validating upgrade/downgrade and idempotency; this migration should have equivalent coverage per project standards for new tools.
   </div>
   
   
   </div>
   
   
   
   
   <small><i>Code Review Run #398665</i></small>
   </div>
   
   ---
   Should Bito avoid suggestions like this for future reviews? (<a href=https://alpha.bito.ai/home/ai-agents/review-rules>Manage Rules</a>)
   - [ ] Yes, avoid them



##########
tests/integration_tests/deletion_retention/prune_audit_tests.py:
##########
@@ -0,0 +1,484 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Integration tests for purge-audit pruning against the real metadata DB.
+
+Rows are seeded directly (pruning reads only ``purge_audit_log``); every
+test's rows carry a distinctive ``entity_type`` prefix and are removed in
+teardown, so no other suite ever sees them. Scoping on ``entity_type``
+rather than ``entity_uuid`` matters: some rows deliberately carry a NULL
+uuid, which a ``LIKE`` on the uuid column would never match.
+"""
+
+from __future__ import annotations
+
+from collections.abc import Callable
+from datetime import timedelta
+from functools import partial
+from typing import Any
+from unittest.mock import patch
+from uuid import UUID, uuid4
+
+import sqlalchemy as sa
+from flask import current_app
+from sqlalchemy.orm import Query
+
+from superset import db
+from superset.commands.deletion_retention import audit, prune_audit
+from superset.commands.deletion_retention.prune_audit import (
+    EVIDENCE_RETENTION_KEY,
+    OPERATIONAL_RETENTION_KEY,
+)
+from superset.models.purge_audit_log import (
+    PurgeAuditLog,
+    STATUS_BLOCKED,
+    STATUS_CONFIRMED,
+    STATUS_FAILED,
+    STATUS_PENDING,
+    STATUS_TARGET_ABSENT,
+)
+from tests.integration_tests.base_tests import SupersetTestCase
+from tests.integration_tests.deletion_retention._base import (
+    ensure_purge_audit_coordination,
+)
+
+_PREFIX: str = "prune_audit_it_"
+_ENTITY_TYPE: str = f"{_PREFIX}slices"
+
+
+class TestPruneAudit(SupersetTestCase):
+    """Behavioral coverage for ``prune_audit.run_prune``."""
+
+    _seq: int
+
+    def setUp(self) -> None:
+        super().setUp()
+        ensure_purge_audit_coordination()
+        self._seq = 0
+        self._cleanup()
+
+    def tearDown(self) -> None:
+        self._cleanup()
+        super().tearDown()
+
+    def _cleanup(self) -> None:
+        db.session.rollback()
+        db.session.execute(
+            sa.delete(PurgeAuditLog.__table__).where(
+                PurgeAuditLog.__table__.c.entity_type == _ENTITY_TYPE
+            )
+        )
+        db.session.commit()
+
+    def add_row(
+        self,
+        status: str,
+        entity: str | None = "e1",
+        age_days: float = 0,
+        trigger: str = audit.TRIGGER_RETENTION,
+    ) -> UUID:
+        """Seed one audit row and return its id.
+
+        Ids rather than instances: an instance whose row a later prune
+        deletes raises ObjectDeletedError on attribute access. ``age_days``
+        may be fractional; a strictly increasing microsecond sequence keeps
+        every ``created_on`` unique so streak ordering is deterministic.
+        ``entity=None`` seeds a row with no ``entity_uuid``.
+        """
+        self._seq += 1
+        row: PurgeAuditLog = PurgeAuditLog(
+            id=uuid4(),
+            status=status,
+            trigger=trigger,
+            actor=audit.ACTOR_SYSTEM,
+            entity_type=_ENTITY_TYPE,
+            entity_uuid=None if entity is None else f"{_PREFIX}{entity}",
+            created_on=audit.utc_now()
+            - timedelta(days=age_days)
+            + timedelta(microseconds=self._seq),
+        )
+        db.session.add(row)
+        db.session.commit()
+        return row.id
+
+    def remaining_ids(self, entity: str | None = "__any__") -> list[UUID]:
+        """Ids of surviving seeded rows, oldest first."""
+        query: Query[PurgeAuditLog] = db.session.query(PurgeAuditLog).filter(
+            PurgeAuditLog.entity_type == _ENTITY_TYPE
+        )
+        if entity != "__any__":
+            query = query.filter(
+                PurgeAuditLog.entity_uuid.is_(None)
+                if entity is None
+                else PurgeAuditLog.entity_uuid == f"{_PREFIX}{entity}"
+            )
+        return [r.id for r in query.order_by(PurgeAuditLog.created_on).all()]
+
+    def run_prune(self, **config: Any) -> prune_audit.PruneRunResult:
+        with patch.dict(current_app.config, config):
+            return prune_audit.run_prune()
+
+    # -- US1: bounded blockage history -------------------------------------
+
+    def test_duplicates_reduce_to_the_earliest_survivor_regardless_of_age(self) -> None:
+        """Keep only the earliest row in a current blockage streak."""
+        ids: list[UUID] = [
+            self.add_row(STATUS_BLOCKED, age_days=5 - i / 100) for i in range(50)
+        ]
+
+        result: prune_audit.PruneRunResult = self.run_prune()
+
+        assert result.blocked_duplicates == 49
+        assert result.carried_over is False
+        assert self.remaining_ids("e1") == [ids[0]]
+
+    def test_backlog_converges_over_bounded_runs(self) -> None:
+        """Drain oversized backlogs across bounded successive runs."""
+        for i in range(56):
+            self.add_row(STATUS_BLOCKED, age_days=3 - i / 1000)
+
+        removed_per_run: list[int] = []
+        with (
+            patch.object(prune_audit, "BATCH_SIZE", 10),
+            patch.object(prune_audit, "MAX_BATCHES_PER_RUN", 2),
+        ):
+            while True:
+                result: prune_audit.PruneRunResult = self.run_prune()
+                removed_per_run.append(result.blocked_duplicates)
+                if not result.carried_over:
+                    break
+
+        assert sum(removed_per_run) == 55  # every duplicate, none of the survivor
+        assert all(n <= 20 for n in removed_per_run)  # never exceeds the budget
+        assert removed_per_run[-1] > 0  # carryover is not reported spuriously
+        assert len(self.remaining_ids("e1")) == 1  # the survivor
+
+    def test_dedup_never_crosses_entities(self) -> None:
+        a_ids: list[UUID] = [
+            self.add_row(STATUS_BLOCKED, entity="a", age_days=2) for _ in range(3)
+        ]
+        b_ids: list[UUID] = [
+            self.add_row(STATUS_BLOCKED, entity="b", age_days=2) for _ in range(4)
+        ]
+
+        result: prune_audit.PruneRunResult = self.run_prune()
+
+        assert result.blocked_duplicates == 5
+        assert self.remaining_ids("a") == [a_ids[0]]
+        assert self.remaining_ids("b") == [b_ids[0]]
+
+    def test_resolved_streak_has_no_survivor_and_ages_out(self) -> None:
+        """Expire blocked rows after newer evidence resolves their streak."""
+        for _ in range(3):
+            self.add_row(STATUS_BLOCKED, age_days=200)
+        confirmed: UUID = self.add_row(STATUS_CONFIRMED, age_days=100)
+        new_survivor: UUID = self.add_row(STATUS_BLOCKED, age_days=50)
+        self.add_row(STATUS_BLOCKED, age_days=40)  # duplicate in the new streak
+
+        result: prune_audit.PruneRunResult = self.run_prune()
+
+        assert result.blocked_duplicates == 1
+        assert result.operational_expired == 3
+        assert set(self.remaining_ids("e1")) == {confirmed, new_survivor}
+
+    def test_a_failed_attempt_does_not_reset_the_blocked_since_survivor(self) -> None:
+        """Keep one blockage streak across a failed purge attempt."""
+        blocked_since: UUID = self.add_row(STATUS_BLOCKED, age_days=200)
+        self.add_row(STATUS_FAILED, age_days=150)  # ages out as operational
+        self.add_row(STATUS_BLOCKED, age_days=100)
+        self.add_row(STATUS_BLOCKED, age_days=50)
+
+        result: prune_audit.PruneRunResult = self.run_prune()
+
+        assert result.blocked_duplicates == 2
+        assert result.operational_expired == 1
+        assert self.remaining_ids("e1") == [blocked_since]
+
+    # -- US2: evidence survives by default ----------------------------------
+
+    def test_defaults_leave_completed_destruction_evidence_untouched(self) -> None:
+        evidence: list[UUID] = [
+            self.add_row(STATUS_CONFIRMED, entity="ev", age_days=3650),
+            self.add_row(STATUS_TARGET_ABSENT, entity="ev", age_days=1000),
+            self.add_row(STATUS_CONFIRMED, entity="ev", age_days=1),
+        ]
+        self.add_row(STATUS_FAILED, entity="ev", age_days=365)  # ages out
+
+        for _ in range(3):  # any sequence of runs (SC-002)
+            result: prune_audit.PruneRunResult = self.run_prune()
+
+        assert result.evidence_expired == 0
+        assert set(self.remaining_ids("ev")) == set(evidence)
+
+    def test_evidence_opt_in_expires_only_rows_older_than_its_window(self) -> None:
+        old: UUID = self.add_row(STATUS_CONFIRMED, entity="ev", age_days=400)
+        young: UUID = self.add_row(STATUS_TARGET_ABSENT, entity="ev", age_days=300)
+
+        result: prune_audit.PruneRunResult = self.run_prune(
+            **{EVIDENCE_RETENTION_KEY: 365}
+        )
+
+        assert result.evidence_expired == 1
+        assert self.remaining_ids("ev") == [young]
+        assert old not in self.remaining_ids("ev")
+
+    def test_opt_in_disabled_again_removes_no_further_evidence(self) -> None:
+        self.add_row(STATUS_CONFIRMED, entity="ev", age_days=400)
+        survivor: UUID = self.add_row(STATUS_CONFIRMED, entity="ev", age_days=390)
+
+        first: prune_audit.PruneRunResult = self.run_prune(
+            **{EVIDENCE_RETENTION_KEY: 395}
+        )
+        assert first.evidence_expired == 1
+
+        second: prune_audit.PruneRunResult = self.run_prune(
+            **{EVIDENCE_RETENTION_KEY: None}
+        )
+        assert second.evidence_expired == 0
+        assert self.remaining_ids("ev") == [survivor]
+
+    def test_pending_and_future_rows_survive_every_configuration(self) -> None:
+        pending: UUID = self.add_row(STATUS_PENDING, entity="px", age_days=3650)
+        future_blocked: UUID = self.add_row(STATUS_BLOCKED, entity="px", age_days=-1)
+        future_confirmed: UUID = self.add_row(
+            STATUS_CONFIRMED, entity="px", age_days=-2
+        )
+
+        result: prune_audit.PruneRunResult = self.run_prune(
+            **{OPERATIONAL_RETENTION_KEY: 1, EVIDENCE_RETENTION_KEY: 1}
+        )
+
+        assert result.total_removed == 0
+        assert set(self.remaining_ids("px")) == {
+            pending,
+            future_blocked,
+            future_confirmed,
+        }
+
+    def test_a_future_dated_outcome_cannot_resolve_a_live_streak(self) -> None:
+        """Exclude future-dated evidence from current streak boundaries."""
+        blocked_since: UUID = self.add_row(STATUS_BLOCKED, entity="sk", age_days=500)
+        self.add_row(STATUS_BLOCKED, entity="sk", age_days=400)
+        skewed: UUID = self.add_row(STATUS_CONFIRMED, entity="sk", age_days=-30)
+
+        result: prune_audit.PruneRunResult = self.run_prune()
+
+        assert result.blocked_duplicates == 1
+        assert result.operational_expired == 0
+        assert set(self.remaining_ids("sk")) == {blocked_since, skewed}
+
+    # -- Survivor invariant under boundary removal (the review's HIGH) ------
+
+    def test_evidence_expiry_spares_a_boundary_that_still_bounds_blocked_rows(
+        self,
+    ) -> None:
+        """Keep evidence while it still bounds surviving blocked rows."""
+        blocked: list[UUID] = [
+            self.add_row(STATUS_BLOCKED, entity="bd", age_days=30) for _ in range(3)
+        ]
+        boundary: UUID = self.add_row(STATUS_CONFIRMED, entity="bd", age_days=20)
+
+        # Evidence window far shorter than the operational one: the boundary
+        # is past its cutoff, the blocked rows it resolved are not.
+        first: prune_audit.PruneRunResult = self.run_prune(
+            **{OPERATIONAL_RETENTION_KEY: 90, EVIDENCE_RETENTION_KEY: 1}
+        )
+
+        assert first.evidence_expired == 0
+        assert set(self.remaining_ids("bd")) == {*blocked, boundary}
+
+        # Once the blocked rows age out, the boundary bounds nothing and
+        # becomes expirable — the guard defers, it does not immortalize.
+        # Categories drain in order within one run, so the blocked rows go
+        # first and the boundary follows in the same pass.
+        second: prune_audit.PruneRunResult = self.run_prune(
+            **{OPERATIONAL_RETENTION_KEY: 10, EVIDENCE_RETENTION_KEY: 1}
+        )
+        assert second.operational_expired == 3
+        assert second.evidence_expired == 1
+        assert self.remaining_ids("bd") == []
+
+    def test_a_block_after_an_unresolved_attempt_is_not_treated_as_a_duplicate(
+        self,
+    ) -> None:
+        """An in-flight attempt is a boundary waiting to happen.
+
+        Finalizing a ``pending`` row resolves it *in place*, keeping its
+        original timestamp — the one way a boundary can appear in the middle
+        of history. The blocked row after it would become the new streak's
+        survivor, so it must not be classified as a duplicate while the
+        attempt is unresolved. Deleting it would destroy the only record
+        that the entity was still blocked after that attempt.
+        """
+        blocked_since: UUID = self.add_row(STATUS_BLOCKED, entity="ua", age_days=100)
+        attempt: UUID = self.add_row(STATUS_PENDING, entity="ua", age_days=50)
+        later_block: UUID = self.add_row(STATUS_BLOCKED, entity="ua", age_days=10)
+
+        result: prune_audit.PruneRunResult = self.run_prune()
+
+        assert result.blocked_duplicates == 0
+        assert set(self.remaining_ids("ua")) == {blocked_since, attempt, later_block}
+
+        # Once the attempt finalizes, the boundary is real: the later block
+        # is the new streak's survivor and stays; the older block is now a
+        # resolved-streak row that ages out on the normal window.
+        db.session.execute(
+            sa.update(PurgeAuditLog.__table__)
+            .where(PurgeAuditLog.__table__.c.id == attempt)
+            .values(status=STATUS_CONFIRMED)
+        )
+        db.session.commit()
+
+        after: prune_audit.PruneRunResult = self.run_prune()
+
+        assert after.blocked_duplicates == 0
+        assert after.operational_expired == 1  # the pre-attempt block
+        assert set(self.remaining_ids("ua")) == {attempt, later_block}
+
+    def test_age_does_not_make_an_unstable_block_expirable(self) -> None:
+        """Keep aged blocked rows whose classification remains unstable.
+
+        Seeded entirely outside the retention window, so only the guard —
+        not the cutoff — can save the row.
+        """
+        blocked_since: UUID = self.add_row(STATUS_BLOCKED, entity="ao", age_days=100)
+        attempt: UUID = self.add_row(STATUS_PENDING, entity="ao", age_days=98)
+        later_block: UUID = self.add_row(STATUS_BLOCKED, entity="ao", age_days=95)
+
+        result: prune_audit.PruneRunResult = self.run_prune()
+
+        assert result.blocked_duplicates == 0
+        assert result.operational_expired == 0
+        assert set(self.remaining_ids("ao")) == {blocked_since, attempt, later_block}
+
+        # Resolving the attempt makes the boundary real: the later block is
+        # the current streak's survivor and is exempt regardless of age,
+        # while the pre-attempt block ages out.
+        db.session.execute(
+            sa.update(PurgeAuditLog.__table__)
+            .where(PurgeAuditLog.__table__.c.id == attempt)
+            .values(status=STATUS_TARGET_ABSENT)
+        )
+        db.session.commit()
+
+        after: prune_audit.PruneRunResult = self.run_prune()
+
+        assert after.operational_expired == 1
+        assert set(self.remaining_ids("ao")) == {attempt, later_block}
+
+    def test_evidence_guard_also_defers_to_an_unresolved_older_attempt(self) -> None:
+        """Treat an older pending attempt as potential blocked evidence."""
+        attempt: UUID = self.add_row(STATUS_PENDING, entity="ug", age_days=400)
+        boundary: UUID = self.add_row(STATUS_CONFIRMED, entity="ug", age_days=300)
+
+        result: prune_audit.PruneRunResult = self.run_prune(
+            **{EVIDENCE_RETENTION_KEY: 200}
+        )
+
+        assert result.evidence_expired == 0
+        assert set(self.remaining_ids("ug")) == {attempt, boundary}
+
+    def test_blocked_rows_without_an_entity_uuid_are_never_aged_out(self) -> None:
+        """Keep UUID-less blocked rows that cannot be proven redundant."""
+        anonymous_block: UUID = self.add_row(STATUS_BLOCKED, entity=None, age_days=1000)
+        self.add_row(STATUS_FAILED, entity=None, age_days=1000)
+
+        result: prune_audit.PruneRunResult = self.run_prune()
+
+        assert result.operational_expired == 1
+        assert self.remaining_ids(None) == [anonymous_block]
+
+    # -- US3: operator controls and observability ---------------------------
+
+    def test_configured_retention_window_is_honored(self) -> None:
+        kept: UUID = self.add_row(STATUS_FAILED, entity="w", age_days=5)
+        self.add_row(STATUS_FAILED, entity="w", age_days=15)
+
+        result: prune_audit.PruneRunResult = self.run_prune(
+            **{OPERATIONAL_RETENTION_KEY: 10}
+        )
+
+        assert result.operational_expired == 1
+        assert self.remaining_ids("w") == [kept]
+
+    def test_invalid_operational_window_skips_the_category_not_widens(self) -> None:
+        """Fail closed when the operational retention window is invalid."""
+        old_failed: UUID = self.add_row(STATUS_FAILED, entity="iv", age_days=1000)
+        self.add_row(STATUS_BLOCKED, entity="iv", age_days=3)
+        self.add_row(STATUS_BLOCKED, entity="iv", age_days=2)
+
+        result: prune_audit.PruneRunResult = self.run_prune(
+            **{OPERATIONAL_RETENTION_KEY: 0}
+        )
+
+        assert result.operational_expired == 0
+        assert result.blocked_duplicates == 1
+        assert result.invalid_config_keys == [OPERATIONAL_RETENTION_KEY]
+        assert old_failed in self.remaining_ids("iv")
+
+    def test_second_run_over_the_same_candidates_removes_and_reports_zero(self) -> None:
+        """Report zero when rerunning over an already-pruned history."""
+        for _ in range(4):
+            self.add_row(STATUS_BLOCKED, entity="cc", age_days=2)
+        self.add_row(STATUS_FAILED, entity="cc", age_days=100)
+
+        first: prune_audit.PruneRunResult = self.run_prune()
+        assert first.blocked_duplicates == 3
+        assert first.operational_expired == 1
+
+        second: prune_audit.PruneRunResult = self.run_prune()
+        assert second.total_removed == 0
+        assert len(self.remaining_ids("cc")) == 1
+
+    def test_a_duplicate_backlog_cannot_starve_the_age_based_categories(self) -> None:
+        """Prevent duplicate backlogs from starving age-based categories."""
+        for _ in range(20):
+            self.add_row(STATUS_BLOCKED, entity="st", age_days=2)
+        old_failed: UUID = self.add_row(STATUS_FAILED, entity="st", age_days=1000)
+
+        with (
+            patch.object(prune_audit, "BATCH_SIZE", 1),
+            patch.object(prune_audit, "MAX_BATCHES_PER_RUN", 2),
+        ):
+            result: prune_audit.PruneRunResult = self.run_prune()
+
+        assert result.blocked_duplicates == 1  # budget-limited, as expected
+        assert result.operational_expired == 1  # but age-out still progressed
+        assert result.carried_over is True
+        assert old_failed not in self.remaining_ids("st")
+
+    def test_delete_rechecks_survivor_after_a_pending_attempt_appears(self) -> None:
+        """Evaluate survivor safety inside the DELETE statement.
+
+        Constructing the candidate query must not freeze its result. A pending
+        attempt committed before execution can become a mid-history boundary,
+        so the later blocked row must remain available as its future survivor.
+        """
+        blocked_since: UUID = self.add_row(STATUS_BLOCKED, entity="rc", age_days=3)
+        later_block: UUID = self.add_row(STATUS_BLOCKED, entity="rc", age_days=1)
+        select_candidates: Callable[[int], sa.sql.Select] = partial(
+            prune_audit._duplicate_candidates, audit.utc_now()

Review Comment:
   <div>
   
   
   <div id="suggestion">
   <div id="issue"><b>Replace private API with public interface</b></div>
   <div id="fix">
   
   Accessing private member `_duplicate_candidates` may break if internal implementation changes; consider using a public interface or clearly document the intentional coupling.
   </div>
   
   
   </div>
   
   
   
   
   <small><i>Code Review Run #398665</i></small>
   </div>
   
   ---
   Should Bito avoid suggestions like this for future reviews? (<a href=https://alpha.bito.ai/home/ai-agents/review-rules>Manage Rules</a>)
   - [ ] Yes, avoid them



##########
tests/unit_tests/commands/deletion_retention/test_prune_audit.py:
##########
@@ -0,0 +1,294 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Unit tests for the pure parts of purge-audit pruning.
+
+Query behavior against real rows (streak survivors, batching, protection
+invariants) is covered by
+``tests/integration_tests/deletion_retention/prune_audit_tests.py``.
+"""
+
+from __future__ import annotations
+
+from contextlib import contextmanager
+from datetime import datetime
+from typing import Any, Iterator
+from unittest.mock import patch
+
+import pytest
+import sqlalchemy as sa
+from flask import current_app
+from sqlalchemy.dialects import mysql
+
+from superset.commands.deletion_retention import prune_audit
+from superset.commands.deletion_retention.prune_audit import (
+    EVIDENCE_RETENTION_KEY,
+    OPERATIONAL_RETENTION_KEY,
+    OPERATIONAL_STATUSES,
+    PROTECTED_STATUSES,
+    PruneRunResult,
+    resolve_evidence_retention_days,
+    resolve_operational_retention_days,
+)
+from superset.models.purge_audit_log import (
+    ALL_STATUSES,
+    STATUS_BLOCKED,
+    STATUS_CONFIRMED,
+    STATUS_FAILED,
+    STATUS_PENDING,
+    STATUS_TARGET_ABSENT,
+)
+
+_METRIC_PREFIX: str = "deletion_retention.prune_purge_audit"
+
+
+@contextmanager
+def without_config(key: str) -> Iterator[None]:
+    """Remove a config key for the duration of the block, then restore it.
+
+    ``patch.dict`` cannot express key *removal*, and the unit-test app
+    fixture is module-scoped, so a bare ``pop`` would leak to later tests.
+    """
+    with patch.dict(current_app.config):
+        current_app.config.pop(key, None)
+        yield
+
+
+def test_retention_categories_partition_every_status() -> None:
+    """Require every status to belong to a retention category.
+
+    A new status added to the model without a category would silently never
+    be pruned; this assertion catches that omission.
+    """
+    assert OPERATIONAL_STATUSES == {STATUS_BLOCKED, STATUS_FAILED}
+    assert PROTECTED_STATUSES == {STATUS_CONFIRMED, STATUS_TARGET_ABSENT}
+    assert not OPERATIONAL_STATUSES & PROTECTED_STATUSES
+    assert OPERATIONAL_STATUSES | PROTECTED_STATUSES | {STATUS_PENDING} == ALL_STATUSES
+
+
+def test_only_proof_of_destruction_breaks_a_blockage_streak() -> None:
+    """Require destruction evidence to break a blockage streak.
+
+    A failed attempt is an infrastructure outcome, and pending is provisional.
+    Neither proves that the blockage cleared.
+    """
+    assert prune_audit._STREAK_BREAKING_STATUSES == {
+        STATUS_CONFIRMED,
+        STATUS_TARGET_ABSENT,
+    }
+    assert STATUS_FAILED not in prune_audit._STREAK_BREAKING_STATUSES
+    assert STATUS_PENDING not in prune_audit._STREAK_BREAKING_STATUSES
+
+
+def test_operational_retention_defaults_to_ninety_days() -> None:
+    """Use the shipped 90-day operational retention default."""
+    assert current_app.config[OPERATIONAL_RETENTION_KEY] == 90
+    assert resolve_operational_retention_days().days == 90
+
+
+@pytest.mark.parametrize("value", [30, 1, 36500])
+def test_operational_retention_accepts_positive_days(value: int) -> None:
+    with patch.dict(current_app.config, {OPERATIONAL_RETENTION_KEY: value}):
+        assert resolve_operational_retention_days() == (value, None)
+
+
+@pytest.mark.parametrize("value", [0, -5, True, False, "ninety", None, 1.5])
+def test_operational_retention_fails_closed_on_invalid_values(value: Any) -> None:
+    """Disable a category and identify its invalid configuration key."""
+    with patch.dict(current_app.config, {OPERATIONAL_RETENTION_KEY: value}):
+        window: prune_audit.ResolvedWindow = resolve_operational_retention_days()
+    assert window.days is None
+    assert window.invalid_key == OPERATIONAL_RETENTION_KEY
+
+
+def test_missing_operational_key_is_reported_as_invalid_not_assumed() -> None:
+    """A popped key is operator error, not a silent 90-day assumption."""
+    with without_config(OPERATIONAL_RETENTION_KEY):
+        window: prune_audit.ResolvedWindow = resolve_operational_retention_days()
+    assert window.days is None
+    assert window.invalid_key == OPERATIONAL_RETENTION_KEY
+
+
+def test_evidence_retention_defaults_to_off_without_warning() -> None:
+    """Unset is the documented never-expire default (FR-006), not an error:
+    disabled, no warning, and no key reported as invalid."""
+    with without_config(EVIDENCE_RETENTION_KEY):
+        with patch.object(prune_audit, "logger") as mock_logger:
+            window: prune_audit.ResolvedWindow = resolve_evidence_retention_days()
+        mock_logger.warning.assert_not_called()
+    assert window == (None, None)
+
+
+def test_evidence_retention_accepts_the_explicit_opt_in() -> None:
+    with patch.dict(current_app.config, {EVIDENCE_RETENTION_KEY: 3650}):
+        window: prune_audit.ResolvedWindow = resolve_evidence_retention_days()
+    assert window.days == 3650
+
+
+@pytest.mark.parametrize("value", [0, -1, True, "forever"])
+def test_evidence_retention_fails_closed_on_invalid_values(value: Any) -> None:
+    with patch.dict(current_app.config, {EVIDENCE_RETENTION_KEY: value}):
+        with patch.object(prune_audit, "logger") as mock_logger:
+            window: prune_audit.ResolvedWindow = resolve_evidence_retention_days()
+        mock_logger.warning.assert_called_once()
+    assert window.days is None
+    assert window.invalid_key == EVIDENCE_RETENTION_KEY
+
+
+def test_prune_run_result_totals_and_dict_shape() -> None:
+    result: PruneRunResult = PruneRunResult(
+        blocked_duplicates=3, operational_expired=2, evidence_expired=1
+    )
+    assert result.total_removed == 6
+    assert result.as_dict() == {
+        "removed": {
+            "blocked_duplicates": 3,
+            "operational_expired": 2,
+            "evidence_expired": 1,
+        },
+        "carried_over": False,
+        "invalid_config_keys": [],
+    }
+
+
+def test_disabled_task_reports_itself_and_removes_nothing() -> None:
+    """Report a disabled run without reaching the prune implementation."""
+    from superset.tasks import deletion_retention as task_module
+
+    with patch.dict(current_app.config, {"PURGE_AUDIT_PRUNING_ENABLED": False}):
+        with (
+            patch.object(task_module, "stats_logger_manager") as mock_stats,
+            patch.object(task_module, "logger") as mock_logger,
+            patch.object(task_module.prune_audit, "run_prune") as mock_run,
+        ):
+            outcome: dict[str, Any] = task_module.prune_purge_audit()
+    assert outcome == {"skipped_disabled": 1}
+    mock_run.assert_not_called()
+    mock_stats.instance.incr.assert_called_once_with(
+        f"{_METRIC_PREFIX}.skipped_disabled"
+    )
+    assert mock_logger.info.called
+
+
+@pytest.mark.parametrize("value", ["false", "0", 1, None])
+def test_non_boolean_master_switch_fails_closed(value: Any) -> None:
+    """Never interpret truthy strings or numeric values as deletion opt-in."""
+    from superset.tasks import deletion_retention as task_module
+
+    with patch.dict(current_app.config, {"PURGE_AUDIT_PRUNING_ENABLED": value}):
+        with (
+            patch.object(task_module, "stats_logger_manager") as mock_stats,
+            patch.object(task_module.prune_audit, "run_prune") as mock_run,
+        ):
+            outcome: dict[str, Any] = task_module.prune_purge_audit()
+
+    assert outcome == {"skipped_invalid_config": 1}
+    mock_run.assert_not_called()
+    mock_stats.instance.incr.assert_called_once_with(
+        f"{_METRIC_PREFIX}.skipped_invalid_config"
+    )
+
+
+def test_failed_run_is_isolated_rolled_back_and_distinguishable() -> None:
+    """Report and isolate a failed pruning run.
+
+    The task rolls back and returns an error marker that cannot be mistaken
+    for a successful run that removed nothing.
+    """
+    from superset.tasks import deletion_retention as task_module
+
+    with patch.dict(current_app.config, {"PURGE_AUDIT_PRUNING_ENABLED": True}):
+        with (
+            patch.object(task_module, "stats_logger_manager") as mock_stats,
+            patch.object(task_module, "db") as mock_db,
+            patch.object(
+                task_module.prune_audit,
+                "run_prune",
+                side_effect=RuntimeError("boom"),
+            ),
+        ):
+            outcome: dict[str, Any] = task_module.prune_purge_audit()
+    assert outcome == {"error": 1}
+    mock_db.session.rollback.assert_called_once()
+    mock_stats.instance.incr.assert_called_once_with(f"{_METRIC_PREFIX}.failed")
+
+
+def test_successful_run_mirrors_counts_and_carryover_into_metrics() -> None:
+    """Expose category counts and convergence through metrics."""
+    from superset.tasks import deletion_retention as task_module
+
+    fake: PruneRunResult = PruneRunResult(
+        blocked_duplicates=7,
+        operational_expired=4,
+        evidence_expired=0,
+        carried_over=True,
+    )

Review Comment:
   <div>
   
   
   <div id="suggestion">
   <div id="issue"><b>Duplicate test setup code</b></div>
   <div id="fix">
   
   The test file contains syntactic duplication (9 lines repeated at lines 238-246 and 264-272) involving identical patch contexts for stats_logger_manager and run_prune. Consider extracting this into a shared fixture to improve maintainability. No changes required from this diff as no modifications are present.
   </div>
   
   
   </div>
   
   
   
   
   <small><i>Code Review Run #398665</i></small>
   </div>
   
   ---
   Should Bito avoid suggestions like this for future reviews? (<a href=https://alpha.bito.ai/home/ai-agents/review-rules>Manage Rules</a>)
   - [ ] Yes, avoid them



##########
tests/integration_tests/deletion_retention/prune_audit_tests.py:
##########
@@ -0,0 +1,484 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Integration tests for purge-audit pruning against the real metadata DB.
+
+Rows are seeded directly (pruning reads only ``purge_audit_log``); every
+test's rows carry a distinctive ``entity_type`` prefix and are removed in
+teardown, so no other suite ever sees them. Scoping on ``entity_type``
+rather than ``entity_uuid`` matters: some rows deliberately carry a NULL
+uuid, which a ``LIKE`` on the uuid column would never match.
+"""
+
+from __future__ import annotations
+
+from collections.abc import Callable
+from datetime import timedelta
+from functools import partial
+from typing import Any
+from unittest.mock import patch
+from uuid import UUID, uuid4
+
+import sqlalchemy as sa
+from flask import current_app
+from sqlalchemy.orm import Query
+
+from superset import db
+from superset.commands.deletion_retention import audit, prune_audit
+from superset.commands.deletion_retention.prune_audit import (
+    EVIDENCE_RETENTION_KEY,
+    OPERATIONAL_RETENTION_KEY,
+)
+from superset.models.purge_audit_log import (
+    PurgeAuditLog,
+    STATUS_BLOCKED,
+    STATUS_CONFIRMED,
+    STATUS_FAILED,
+    STATUS_PENDING,
+    STATUS_TARGET_ABSENT,
+)
+from tests.integration_tests.base_tests import SupersetTestCase
+from tests.integration_tests.deletion_retention._base import (
+    ensure_purge_audit_coordination,
+)
+
+_PREFIX: str = "prune_audit_it_"
+_ENTITY_TYPE: str = f"{_PREFIX}slices"
+
+
+class TestPruneAudit(SupersetTestCase):
+    """Behavioral coverage for ``prune_audit.run_prune``."""
+
+    _seq: int
+
+    def setUp(self) -> None:
+        super().setUp()
+        ensure_purge_audit_coordination()
+        self._seq = 0
+        self._cleanup()
+
+    def tearDown(self) -> None:
+        self._cleanup()
+        super().tearDown()
+
+    def _cleanup(self) -> None:
+        db.session.rollback()
+        db.session.execute(
+            sa.delete(PurgeAuditLog.__table__).where(
+                PurgeAuditLog.__table__.c.entity_type == _ENTITY_TYPE
+            )
+        )
+        db.session.commit()
+
+    def add_row(
+        self,
+        status: str,
+        entity: str | None = "e1",
+        age_days: float = 0,
+        trigger: str = audit.TRIGGER_RETENTION,
+    ) -> UUID:
+        """Seed one audit row and return its id.
+
+        Ids rather than instances: an instance whose row a later prune
+        deletes raises ObjectDeletedError on attribute access. ``age_days``
+        may be fractional; a strictly increasing microsecond sequence keeps
+        every ``created_on`` unique so streak ordering is deterministic.
+        ``entity=None`` seeds a row with no ``entity_uuid``.
+        """
+        self._seq += 1
+        row: PurgeAuditLog = PurgeAuditLog(
+            id=uuid4(),
+            status=status,
+            trigger=trigger,
+            actor=audit.ACTOR_SYSTEM,
+            entity_type=_ENTITY_TYPE,
+            entity_uuid=None if entity is None else f"{_PREFIX}{entity}",
+            created_on=audit.utc_now()
+            - timedelta(days=age_days)
+            + timedelta(microseconds=self._seq),
+        )
+        db.session.add(row)
+        db.session.commit()
+        return row.id
+
+    def remaining_ids(self, entity: str | None = "__any__") -> list[UUID]:
+        """Ids of surviving seeded rows, oldest first."""
+        query: Query[PurgeAuditLog] = db.session.query(PurgeAuditLog).filter(
+            PurgeAuditLog.entity_type == _ENTITY_TYPE
+        )
+        if entity != "__any__":
+            query = query.filter(
+                PurgeAuditLog.entity_uuid.is_(None)
+                if entity is None
+                else PurgeAuditLog.entity_uuid == f"{_PREFIX}{entity}"
+            )
+        return [r.id for r in query.order_by(PurgeAuditLog.created_on).all()]
+
+    def run_prune(self, **config: Any) -> prune_audit.PruneRunResult:
+        with patch.dict(current_app.config, config):
+            return prune_audit.run_prune()
+
+    # -- US1: bounded blockage history -------------------------------------
+
+    def test_duplicates_reduce_to_the_earliest_survivor_regardless_of_age(self) -> None:
+        """Keep only the earliest row in a current blockage streak."""
+        ids: list[UUID] = [
+            self.add_row(STATUS_BLOCKED, age_days=5 - i / 100) for i in range(50)
+        ]
+
+        result: prune_audit.PruneRunResult = self.run_prune()
+
+        assert result.blocked_duplicates == 49
+        assert result.carried_over is False
+        assert self.remaining_ids("e1") == [ids[0]]
+
+    def test_backlog_converges_over_bounded_runs(self) -> None:
+        """Drain oversized backlogs across bounded successive runs."""
+        for i in range(56):
+            self.add_row(STATUS_BLOCKED, age_days=3 - i / 1000)
+
+        removed_per_run: list[int] = []
+        with (
+            patch.object(prune_audit, "BATCH_SIZE", 10),
+            patch.object(prune_audit, "MAX_BATCHES_PER_RUN", 2),
+        ):
+            while True:
+                result: prune_audit.PruneRunResult = self.run_prune()
+                removed_per_run.append(result.blocked_duplicates)
+                if not result.carried_over:
+                    break
+
+        assert sum(removed_per_run) == 55  # every duplicate, none of the survivor
+        assert all(n <= 20 for n in removed_per_run)  # never exceeds the budget
+        assert removed_per_run[-1] > 0  # carryover is not reported spuriously
+        assert len(self.remaining_ids("e1")) == 1  # the survivor
+
+    def test_dedup_never_crosses_entities(self) -> None:
+        a_ids: list[UUID] = [
+            self.add_row(STATUS_BLOCKED, entity="a", age_days=2) for _ in range(3)
+        ]
+        b_ids: list[UUID] = [
+            self.add_row(STATUS_BLOCKED, entity="b", age_days=2) for _ in range(4)
+        ]
+
+        result: prune_audit.PruneRunResult = self.run_prune()
+
+        assert result.blocked_duplicates == 5
+        assert self.remaining_ids("a") == [a_ids[0]]
+        assert self.remaining_ids("b") == [b_ids[0]]
+
+    def test_resolved_streak_has_no_survivor_and_ages_out(self) -> None:
+        """Expire blocked rows after newer evidence resolves their streak."""
+        for _ in range(3):
+            self.add_row(STATUS_BLOCKED, age_days=200)
+        confirmed: UUID = self.add_row(STATUS_CONFIRMED, age_days=100)
+        new_survivor: UUID = self.add_row(STATUS_BLOCKED, age_days=50)
+        self.add_row(STATUS_BLOCKED, age_days=40)  # duplicate in the new streak
+
+        result: prune_audit.PruneRunResult = self.run_prune()
+
+        assert result.blocked_duplicates == 1
+        assert result.operational_expired == 3
+        assert set(self.remaining_ids("e1")) == {confirmed, new_survivor}
+
+    def test_a_failed_attempt_does_not_reset_the_blocked_since_survivor(self) -> None:
+        """Keep one blockage streak across a failed purge attempt."""
+        blocked_since: UUID = self.add_row(STATUS_BLOCKED, age_days=200)
+        self.add_row(STATUS_FAILED, age_days=150)  # ages out as operational
+        self.add_row(STATUS_BLOCKED, age_days=100)
+        self.add_row(STATUS_BLOCKED, age_days=50)
+
+        result: prune_audit.PruneRunResult = self.run_prune()
+
+        assert result.blocked_duplicates == 2
+        assert result.operational_expired == 1
+        assert self.remaining_ids("e1") == [blocked_since]
+
+    # -- US2: evidence survives by default ----------------------------------
+
+    def test_defaults_leave_completed_destruction_evidence_untouched(self) -> None:
+        evidence: list[UUID] = [
+            self.add_row(STATUS_CONFIRMED, entity="ev", age_days=3650),
+            self.add_row(STATUS_TARGET_ABSENT, entity="ev", age_days=1000),
+            self.add_row(STATUS_CONFIRMED, entity="ev", age_days=1),
+        ]
+        self.add_row(STATUS_FAILED, entity="ev", age_days=365)  # ages out
+
+        for _ in range(3):  # any sequence of runs (SC-002)
+            result: prune_audit.PruneRunResult = self.run_prune()
+
+        assert result.evidence_expired == 0
+        assert set(self.remaining_ids("ev")) == set(evidence)
+
+    def test_evidence_opt_in_expires_only_rows_older_than_its_window(self) -> None:
+        old: UUID = self.add_row(STATUS_CONFIRMED, entity="ev", age_days=400)
+        young: UUID = self.add_row(STATUS_TARGET_ABSENT, entity="ev", age_days=300)
+
+        result: prune_audit.PruneRunResult = self.run_prune(
+            **{EVIDENCE_RETENTION_KEY: 365}
+        )
+
+        assert result.evidence_expired == 1
+        assert self.remaining_ids("ev") == [young]
+        assert old not in self.remaining_ids("ev")
+
+    def test_opt_in_disabled_again_removes_no_further_evidence(self) -> None:
+        self.add_row(STATUS_CONFIRMED, entity="ev", age_days=400)
+        survivor: UUID = self.add_row(STATUS_CONFIRMED, entity="ev", age_days=390)
+
+        first: prune_audit.PruneRunResult = self.run_prune(
+            **{EVIDENCE_RETENTION_KEY: 395}
+        )
+        assert first.evidence_expired == 1
+
+        second: prune_audit.PruneRunResult = self.run_prune(
+            **{EVIDENCE_RETENTION_KEY: None}
+        )
+        assert second.evidence_expired == 0
+        assert self.remaining_ids("ev") == [survivor]
+
+    def test_pending_and_future_rows_survive_every_configuration(self) -> None:
+        pending: UUID = self.add_row(STATUS_PENDING, entity="px", age_days=3650)
+        future_blocked: UUID = self.add_row(STATUS_BLOCKED, entity="px", age_days=-1)
+        future_confirmed: UUID = self.add_row(
+            STATUS_CONFIRMED, entity="px", age_days=-2
+        )
+
+        result: prune_audit.PruneRunResult = self.run_prune(
+            **{OPERATIONAL_RETENTION_KEY: 1, EVIDENCE_RETENTION_KEY: 1}
+        )
+
+        assert result.total_removed == 0
+        assert set(self.remaining_ids("px")) == {
+            pending,
+            future_blocked,
+            future_confirmed,
+        }
+
+    def test_a_future_dated_outcome_cannot_resolve_a_live_streak(self) -> None:
+        """Exclude future-dated evidence from current streak boundaries."""
+        blocked_since: UUID = self.add_row(STATUS_BLOCKED, entity="sk", age_days=500)
+        self.add_row(STATUS_BLOCKED, entity="sk", age_days=400)
+        skewed: UUID = self.add_row(STATUS_CONFIRMED, entity="sk", age_days=-30)
+
+        result: prune_audit.PruneRunResult = self.run_prune()
+
+        assert result.blocked_duplicates == 1
+        assert result.operational_expired == 0
+        assert set(self.remaining_ids("sk")) == {blocked_since, skewed}
+
+    # -- Survivor invariant under boundary removal (the review's HIGH) ------
+
+    def test_evidence_expiry_spares_a_boundary_that_still_bounds_blocked_rows(
+        self,
+    ) -> None:
+        """Keep evidence while it still bounds surviving blocked rows."""
+        blocked: list[UUID] = [
+            self.add_row(STATUS_BLOCKED, entity="bd", age_days=30) for _ in range(3)
+        ]
+        boundary: UUID = self.add_row(STATUS_CONFIRMED, entity="bd", age_days=20)
+
+        # Evidence window far shorter than the operational one: the boundary
+        # is past its cutoff, the blocked rows it resolved are not.
+        first: prune_audit.PruneRunResult = self.run_prune(
+            **{OPERATIONAL_RETENTION_KEY: 90, EVIDENCE_RETENTION_KEY: 1}
+        )
+
+        assert first.evidence_expired == 0
+        assert set(self.remaining_ids("bd")) == {*blocked, boundary}
+
+        # Once the blocked rows age out, the boundary bounds nothing and
+        # becomes expirable — the guard defers, it does not immortalize.
+        # Categories drain in order within one run, so the blocked rows go
+        # first and the boundary follows in the same pass.
+        second: prune_audit.PruneRunResult = self.run_prune(
+            **{OPERATIONAL_RETENTION_KEY: 10, EVIDENCE_RETENTION_KEY: 1}
+        )
+        assert second.operational_expired == 3
+        assert second.evidence_expired == 1
+        assert self.remaining_ids("bd") == []
+
+    def test_a_block_after_an_unresolved_attempt_is_not_treated_as_a_duplicate(
+        self,
+    ) -> None:
+        """An in-flight attempt is a boundary waiting to happen.
+
+        Finalizing a ``pending`` row resolves it *in place*, keeping its
+        original timestamp — the one way a boundary can appear in the middle
+        of history. The blocked row after it would become the new streak's
+        survivor, so it must not be classified as a duplicate while the
+        attempt is unresolved. Deleting it would destroy the only record
+        that the entity was still blocked after that attempt.
+        """
+        blocked_since: UUID = self.add_row(STATUS_BLOCKED, entity="ua", age_days=100)
+        attempt: UUID = self.add_row(STATUS_PENDING, entity="ua", age_days=50)
+        later_block: UUID = self.add_row(STATUS_BLOCKED, entity="ua", age_days=10)
+
+        result: prune_audit.PruneRunResult = self.run_prune()
+
+        assert result.blocked_duplicates == 0
+        assert set(self.remaining_ids("ua")) == {blocked_since, attempt, later_block}
+
+        # Once the attempt finalizes, the boundary is real: the later block
+        # is the new streak's survivor and stays; the older block is now a
+        # resolved-streak row that ages out on the normal window.
+        db.session.execute(
+            sa.update(PurgeAuditLog.__table__)
+            .where(PurgeAuditLog.__table__.c.id == attempt)
+            .values(status=STATUS_CONFIRMED)
+        )
+        db.session.commit()
+
+        after: prune_audit.PruneRunResult = self.run_prune()
+
+        assert after.blocked_duplicates == 0
+        assert after.operational_expired == 1  # the pre-attempt block
+        assert set(self.remaining_ids("ua")) == {attempt, later_block}
+
+    def test_age_does_not_make_an_unstable_block_expirable(self) -> None:
+        """Keep aged blocked rows whose classification remains unstable.
+
+        Seeded entirely outside the retention window, so only the guard —
+        not the cutoff — can save the row.
+        """
+        blocked_since: UUID = self.add_row(STATUS_BLOCKED, entity="ao", age_days=100)
+        attempt: UUID = self.add_row(STATUS_PENDING, entity="ao", age_days=98)
+        later_block: UUID = self.add_row(STATUS_BLOCKED, entity="ao", age_days=95)
+
+        result: prune_audit.PruneRunResult = self.run_prune()
+
+        assert result.blocked_duplicates == 0
+        assert result.operational_expired == 0
+        assert set(self.remaining_ids("ao")) == {blocked_since, attempt, later_block}
+
+        # Resolving the attempt makes the boundary real: the later block is
+        # the current streak's survivor and is exempt regardless of age,
+        # while the pre-attempt block ages out.
+        db.session.execute(
+            sa.update(PurgeAuditLog.__table__)
+            .where(PurgeAuditLog.__table__.c.id == attempt)
+            .values(status=STATUS_TARGET_ABSENT)
+        )
+        db.session.commit()
+
+        after: prune_audit.PruneRunResult = self.run_prune()
+
+        assert after.operational_expired == 1
+        assert set(self.remaining_ids("ao")) == {attempt, later_block}
+
+    def test_evidence_guard_also_defers_to_an_unresolved_older_attempt(self) -> None:
+        """Treat an older pending attempt as potential blocked evidence."""
+        attempt: UUID = self.add_row(STATUS_PENDING, entity="ug", age_days=400)
+        boundary: UUID = self.add_row(STATUS_CONFIRMED, entity="ug", age_days=300)
+
+        result: prune_audit.PruneRunResult = self.run_prune(
+            **{EVIDENCE_RETENTION_KEY: 200}
+        )
+
+        assert result.evidence_expired == 0
+        assert set(self.remaining_ids("ug")) == {attempt, boundary}
+
+    def test_blocked_rows_without_an_entity_uuid_are_never_aged_out(self) -> None:
+        """Keep UUID-less blocked rows that cannot be proven redundant."""
+        anonymous_block: UUID = self.add_row(STATUS_BLOCKED, entity=None, age_days=1000)
+        self.add_row(STATUS_FAILED, entity=None, age_days=1000)
+
+        result: prune_audit.PruneRunResult = self.run_prune()
+
+        assert result.operational_expired == 1
+        assert self.remaining_ids(None) == [anonymous_block]
+
+    # -- US3: operator controls and observability ---------------------------
+
+    def test_configured_retention_window_is_honored(self) -> None:
+        kept: UUID = self.add_row(STATUS_FAILED, entity="w", age_days=5)
+        self.add_row(STATUS_FAILED, entity="w", age_days=15)
+
+        result: prune_audit.PruneRunResult = self.run_prune(
+            **{OPERATIONAL_RETENTION_KEY: 10}
+        )
+
+        assert result.operational_expired == 1
+        assert self.remaining_ids("w") == [kept]
+
+    def test_invalid_operational_window_skips_the_category_not_widens(self) -> None:
+        """Fail closed when the operational retention window is invalid."""
+        old_failed: UUID = self.add_row(STATUS_FAILED, entity="iv", age_days=1000)
+        self.add_row(STATUS_BLOCKED, entity="iv", age_days=3)
+        self.add_row(STATUS_BLOCKED, entity="iv", age_days=2)
+
+        result: prune_audit.PruneRunResult = self.run_prune(
+            **{OPERATIONAL_RETENTION_KEY: 0}
+        )
+
+        assert result.operational_expired == 0
+        assert result.blocked_duplicates == 1
+        assert result.invalid_config_keys == [OPERATIONAL_RETENTION_KEY]
+        assert old_failed in self.remaining_ids("iv")
+
+    def test_second_run_over_the_same_candidates_removes_and_reports_zero(self) -> None:
+        """Report zero when rerunning over an already-pruned history."""
+        for _ in range(4):
+            self.add_row(STATUS_BLOCKED, entity="cc", age_days=2)
+        self.add_row(STATUS_FAILED, entity="cc", age_days=100)
+
+        first: prune_audit.PruneRunResult = self.run_prune()
+        assert first.blocked_duplicates == 3
+        assert first.operational_expired == 1
+
+        second: prune_audit.PruneRunResult = self.run_prune()
+        assert second.total_removed == 0
+        assert len(self.remaining_ids("cc")) == 1
+
+    def test_a_duplicate_backlog_cannot_starve_the_age_based_categories(self) -> None:
+        """Prevent duplicate backlogs from starving age-based categories."""
+        for _ in range(20):
+            self.add_row(STATUS_BLOCKED, entity="st", age_days=2)
+        old_failed: UUID = self.add_row(STATUS_FAILED, entity="st", age_days=1000)
+
+        with (
+            patch.object(prune_audit, "BATCH_SIZE", 1),
+            patch.object(prune_audit, "MAX_BATCHES_PER_RUN", 2),
+        ):
+            result: prune_audit.PruneRunResult = self.run_prune()
+
+        assert result.blocked_duplicates == 1  # budget-limited, as expected
+        assert result.operational_expired == 1  # but age-out still progressed
+        assert result.carried_over is True
+        assert old_failed not in self.remaining_ids("st")
+
+    def test_delete_rechecks_survivor_after_a_pending_attempt_appears(self) -> None:
+        """Evaluate survivor safety inside the DELETE statement.
+
+        Constructing the candidate query must not freeze its result. A pending
+        attempt committed before execution can become a mid-history boundary,
+        so the later blocked row must remain available as its future survivor.
+        """
+        blocked_since: UUID = self.add_row(STATUS_BLOCKED, entity="rc", age_days=3)
+        later_block: UUID = self.add_row(STATUS_BLOCKED, entity="rc", age_days=1)
+        select_candidates: Callable[[int], sa.sql.Select] = partial(
+            prune_audit._duplicate_candidates, audit.utc_now()
+        )
+
+        attempt: UUID = self.add_row(STATUS_PENDING, entity="rc", age_days=2)
+        removed: int = prune_audit._delete_batch(select_candidates)

Review Comment:
   <div>
   
   
   <div id="suggestion">
   <div id="issue"><b>Replace private API with public interface</b></div>
   <div id="fix">
   
   Accessing private member `_delete_batch` may break if internal implementation changes; consider using a public interface or clearly document the intentional coupling.
   </div>
   
   
   </div>
   
   
   
   
   <small><i>Code Review Run #398665</i></small>
   </div>
   
   ---
   Should Bito avoid suggestions like this for future reviews? (<a href=https://alpha.bito.ai/home/ai-agents/review-rules>Manage Rules</a>)
   - [ ] Yes, avoid them



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org


Re: [PR] feat(deletion-retention): scheduled pruning policy for the purge audit log [superset]

Posted by "mikebridge (via GitHub)" <gi...@apache.org>.
mikebridge commented on code in PR #43490:
URL: https://github.com/apache/superset/pull/43490#discussion_r3978218473


##########
superset/commands/deletion_retention/prune_audit.py:
##########
@@ -0,0 +1,726 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Deletion-only pruning of the ``purge_audit_log`` table.
+
+Bounds the audit history's growth without ever weakening its evidentiary
+value. Three delete categories, applied in priority order under one shared
+per-run batch budget:
+
+1. **Blocked duplicates** — within an entity's *current* blockage streak
+   (its ``blocked`` rows newer than the entity's newest streak-breaking
+   row), only the first row of each run of consecutive same-reason rows
+   survives: the streak's earliest row and the first row after every
+   change of block reason. Later same-reason repeats are removed
+   regardless of age. The survivors carry the "blocked since" fact and
+   the reason history — for coded reasons, the same rows the audit writer's
+   own suppression rule retains (reason-less legacy runs are additionally
+   collapsed to their earliest here) — and are never deleted while the
+   streak is current.
+2. **Operational expiry** — ``blocked`` rows of *resolved* streaks and
+   ``failed`` rows older than ``PURGE_AUDIT_OPERATIONAL_RETENTION_DAYS``
+   age out.
+3. **Evidence expiry** — ``confirmed`` / ``target_absent`` rows are
+   untouchable unless ``PURGE_AUDIT_EVIDENCE_RETENTION_DAYS`` is
+   explicitly set (the operator's compliance assertion), and then only
+   rows older than that window.
+
+``pending`` rows belong to :func:`audit.reconcile_pending` and rows with
+future timestamps (clock skew) are excluded from every category *and* from
+streak classification, so a skewed writer cannot reclassify a live streak.
+
+Three invariants keep the survivor safe without a distributed lock, which
+matters because runs can overlap and both ``reconcile_pending`` and the
+purge path finalize rows concurrently. A duplicate is only ever deleted
+when no concurrent transition could turn it into a survivor first:
+
+* **A boundary is never removed while it still bounds anything.** Evidence
+  expiry refuses to delete a row while an older ``blocked`` or ``pending``
+  row for the same entity survives. Boundaries therefore never *recede*,
+  which would otherwise promote resolved rows into a current streak.
+* **A blocked row whose classification is unstable is never deleted.**
+  Finalizing a ``pending`` row resolves it *in place*, keeping its original
+  timestamp, so an unresolved attempt is a boundary that may appear
+  mid-history at any moment — and the blocked row after it would become its
+  streak's survivor. Blocked rows preceded by an unresolved attempt are
+  therefore skipped by **both** the duplicate and the operational category.
+  Age does not stabilize the classification: an old blocked row inside a
+  live streak is exactly the "blocked for years" case FR-009 protects.
+  Without this, a boundary moving *forward* would demote the current
+  survivor and promote the next row into its place — possibly a row already
+  selected for deletion.
+* **Deletes are conditional and counted from rowcounts.** A row whose
+  status changed since selection is not matched, so overlapping runs can
+  neither double-remove nor double-report.
+
+Boundaries moving forward past *all* of an entity's blocked rows leave no
+survivor to promote, so a selected duplicate may be removed slightly ahead
+of its retention window in that case. It was redundant either way and the
+streak's earliest row is untouched.
+
+Timestamps order rows. Ties are possible — legacy second-precision rows,
+two writers within one clock tick — and are resolved on the preserving
+side: a ``pending`` row tied with a blocked row counts as preceding it (the
+block is deferred until the attempt resolves); a blocked row tied with a
+boundary sits on the boundary's *resolved* side (it ages out instead of
+seeding a new current streak, and the boundary is not removed before it);
+and tied same-reason blocked rows are all retained.
+
+Candidate selection is embedded in each ``DELETE`` statement. The derived-table
+wrapper keeps that shape legal on MySQL while ensuring that a pending or
+recovered row committed before the delete is evaluated participates in the
+survivor and boundary predicates. No stale list of candidate ids crosses a
+transaction boundary.
+
+Audit creation/recovery and every pruning batch take the same singleton
+database write lock before assigning a timestamp or evaluating candidates.
+The lock is held through commit, so an audit row cannot become visible in an
+already-processed logical past. Automatic pruning still ships disabled by
+default so operators explicitly choose their retention policy.
+"""
+
+from __future__ import annotations
+
+import logging
+from collections.abc import Callable
+from dataclasses import dataclass, field
+from datetime import datetime, timedelta
+from functools import partial
+from typing import Any, Literal, NamedTuple, TypeAlias
+
+import sqlalchemy as sa
+from flask import current_app
+
+from superset import db
+from superset.commands.deletion_retention.audit import (
+    acquire_coordination_lock,
+    TRIGGER_FORCE,
+    utc_now,
+)
+from superset.models.purge_audit_log import (
+    PurgeAuditLog,
+    STATUS_BLOCKED,
+    STATUS_CONFIRMED,
+    STATUS_FAILED,
+    STATUS_PENDING,
+    STATUS_TARGET_ABSENT,
+)
+
+logger: logging.Logger = logging.getLogger(__name__)
+
+#: Operational records: noise-prone outcomes whose compliance value decays —
+#: a blocked or failed purge leaves the object in place (FR-001).
+OPERATIONAL_STATUSES: frozenset[str] = frozenset({STATUS_BLOCKED, STATUS_FAILED})
+#: Protected evidence: the only surviving trace of a destroyed object.
+PROTECTED_STATUSES: frozenset[str] = frozenset({STATUS_CONFIRMED, STATUS_TARGET_ABSENT})
+#: The outcomes that end a blockage streak — proof the object is gone.
+#:
+#: ``failed`` is deliberately absent. A failed purge is an infrastructure
+#: outcome (the cascade raised), not evidence the blockage cleared: the
+#: policy that blocked the entity is untouched, so the blockage continues
+#: across it. Treating ``failed`` as a boundary would let one transient
+#: error demote the "blocked since" survivor to an ageing duplicate and
+#: restate the blockage as beginning after the failure — losing exactly the
+#: fact FR-003 exists to preserve. ``pending`` is provisional and likewise
+#: neither joins nor breaks streaks.
+_STREAK_BREAKING_STATUSES: frozenset[str] = frozenset(
+    {STATUS_CONFIRMED, STATUS_TARGET_ABSENT}
+)
+
+#: Rows deleted per statement, matching the purge task's batch convention.
+BATCH_SIZE: int = 500
+#: One shared budget for the whole run across all three categories; the
+#: remaining backlog carries over to the next scheduled run (FR-004/SC-004).
+MAX_BATCHES_PER_RUN: int = 10
+
+OPERATIONAL_RETENTION_KEY: str = "PURGE_AUDIT_OPERATIONAL_RETENTION_DAYS"
+EVIDENCE_RETENTION_KEY: str = "PURGE_AUDIT_EVIDENCE_RETENTION_DAYS"
+
+
+class ResolvedWindow(NamedTuple):
+    """A retention window resolved from config.
+
+    Distinguishes the three outcomes a caller must tell apart: a usable
+    window, the deliberate "off" default, and operator error. Collapsing
+    the last two into a bare ``None`` would force every caller to re-read
+    config to find out which it got.
+    """
+
+    days: int | None
+    invalid_key: str | None = None
+
+
+def _validated_window(key: str, value: Any) -> ResolvedWindow:
+    """Validate a configured day count, failing closed on anything odd.
+
+    An invalid value disables its category for the run rather than widening
+    removal (FR-005/SC-005).
+    """
+    # bool is an int subclass and floats would silently truncate — both are
+    # config mistakes, not day counts, on a knob that deletes rows.
+    if not isinstance(value, bool) and not isinstance(value, float):
+        try:
+            days: int = int(value)
+        except (TypeError, ValueError):
+            days = 0
+        if days > 0:
+            return ResolvedWindow(days)
+    logger.warning(
+        "prune_audit: invalid %s=%r; skipping this category for the run "
+        "(pruning never widens on bad configuration)",
+        key,
+        value,
+    )
+    return ResolvedWindow(None, key)
+
+
+def resolve_operational_retention_days() -> ResolvedWindow:
+    """The operational retention window, or a disabled window when invalid."""
+    return _validated_window(
+        OPERATIONAL_RETENTION_KEY, current_app.config.get(OPERATIONAL_RETENTION_KEY)
+    )
+
+
+def resolve_evidence_retention_days() -> ResolvedWindow:
+    """The evidence expiration window; disabled unless explicitly opted in.
+
+    Unset is the documented "never expire evidence" default (FR-006), not an
+    error, so it produces a disabled window with no warning.
+    """
+    value: Any = current_app.config.get(EVIDENCE_RETENTION_KEY)
+    if value is None:
+        return ResolvedWindow(None)
+    return _validated_window(EVIDENCE_RETENTION_KEY, value)
+
+
+@dataclass
+class PruneRunResult:
+    """Per-category removal counts and run disposition for one pruning run."""
+
+    blocked_duplicates: int = 0
+    operational_expired: int = 0
+    evidence_expired: int = 0
+    #: True when the shared batch budget ran out before every category's
+    #: candidates were drained; the remainder converges on later runs.
+    carried_over: bool = False
+    invalid_config_keys: list[str] = field(default_factory=list)
+
+    @property
+    def total_removed(self) -> int:
+        """Total rows removed across every category."""
+        return (
+            self.blocked_duplicates + self.operational_expired + self.evidence_expired
+        )
+
+    def as_dict(self) -> dict[str, Any]:
+        """The task-return / log-line shape of this result."""
+        return {
+            "removed": {
+                "blocked_duplicates": self.blocked_duplicates,
+                "operational_expired": self.operational_expired,
+                "evidence_expired": self.evidence_expired,
+            },
+            "carried_over": self.carried_over,
+            "invalid_config_keys": list(self.invalid_config_keys),
+        }
+
+
+def _streak_boundary_subquery(now: datetime) -> sa.Subquery:
+    """Per entity, the ``created_on`` of its newest streak-breaking row.
+
+    Entities absent from this subquery have never been proven destroyed —
+    all their blocked rows form one current streak. Future-dated rows are
+    excluded so a skewed writer clock cannot push the boundary ahead of a
+    live streak and make its rows look resolved.
+    """
+    table: sa.Table = PurgeAuditLog.__table__
+    return (
+        sa.select(
+            table.c.entity_type.label("entity_type"),
+            table.c.entity_uuid.label("entity_uuid"),
+            sa.func.max(table.c.created_on).label("boundary"),
+        )
+        .where(table.c.status.in_(_STREAK_BREAKING_STATUSES))
+        .where(table.c.entity_uuid.is_not(None))
+        .where(table.c.created_on <= now)
+        .group_by(table.c.entity_type, table.c.entity_uuid)
+        .subquery("streak_boundary")
+    )
+
+
+def _with_boundary(table: sa.Table, boundary: sa.Subquery) -> sa.Join:
+    """Outer-join each row to its entity's streak boundary (NULL if none)."""
+    return table.outerjoin(
+        boundary,
+        sa.and_(
+            table.c.entity_type == boundary.c.entity_type,
+            table.c.entity_uuid == boundary.c.entity_uuid,
+        ),
+    )
+
+
+def _in_current_streak(
+    row: sa.FromClause, boundary: sa.Subquery
+) -> sa.ColumnElement[bool]:
+    """Whether ``row`` is newer than its entity's boundary (or there is none).
+
+    Strictly newer: a row tied with the boundary sits on its resolved side.
+    A boundary proves the object was gone at that instant, and a block that
+    cannot be ordered after the destruction must not seed a new "current"
+    streak — that would mint a survivor exempt from age-out forever for an
+    object that no longer exists.
+    """
+    return sa.or_(boundary.c.boundary.is_(None), row.c.created_on > boundary.c.boundary)
+
+
+def _repeats_an_earlier_block(
+    table: sa.Table, boundary: sa.Subquery
+) -> sa.ColumnElement[bool]:
+    """Whether a same-reason current-streak block precedes this row with no
+    change of reason in between.
+
+    This is the audit writer's suppression rule
+    (:func:`audit.finalize_retention_blocked`) applied retroactively: a
+    block repeating the reason of the block just before it adds nothing,
+    while the first block after a reason change is the only durable record
+    of the new cause and is retained. The comparison is NULL-safe, so
+    consecutive reason-less (pre-feature) rows count as one run, deduped to
+    the run's earliest, and the first coded block ends that run. Note this
+    is *stricter* than the writer for reason-less rows: the writer never
+    suppresses a reason-less block (``_suppress_redundant_block`` bails on a
+    missing code), so here the pruner additionally collapses legacy
+    pre-feature duplicates — the streak's earliest "blocked since" row is
+    still always kept. Tied same-reason rows are not "earlier" than each
+    other, so all of them are kept.
+
+    A ``force`` row is exempt: it is never reported as a repeat (see the
+    return), so it is kept out of the duplicate category and marked a
+    survivor while its streak is current. It is also exempt from operational
+    age-out (see :func:`_operational_candidates`), so an operator force-purge
+    block is retained permanently — never pruned by either category.
+    """
+    earlier: sa.FromClause = table.alias("earlier_block")
+    between: sa.FromClause = table.alias("reason_change")
+    reason_changed_between: sa.ColumnElement[bool] = sa.exists(
+        sa.select(sa.literal(1))
+        .select_from(between)
+        .where(
+            sa.and_(
+                between.c.status == STATUS_BLOCKED,
+                between.c.entity_type == table.c.entity_type,
+                between.c.entity_uuid == table.c.entity_uuid,
+                # Inclusive bounds: a differing-reason block sharing an
+                # exact timestamp with either endpoint still breaks the run,
+                # so a reason-transition row tied with a neighbour is
+                # preserved as a run head rather than pruned as a repeat
+                # (the same preserving-side tie rule the pending and evidence
+                # guards use). Inclusive bounds only ever add boundaries —
+                # i.e. only ever preserve more, never delete more.
+                between.c.created_on >= earlier.c.created_on,
+                between.c.created_on <= table.c.created_on,
+                between.c.reason.is_distinct_from(table.c.reason),
+            )
+        )
+        .correlate(table, earlier)
+    )
+    repeats: sa.ColumnElement[bool] = sa.exists(
+        sa.select(sa.literal(1))
+        .select_from(earlier)
+        .where(
+            sa.and_(
+                earlier.c.status == STATUS_BLOCKED,
+                earlier.c.entity_type == table.c.entity_type,
+                earlier.c.entity_uuid == table.c.entity_uuid,
+                _in_current_streak(earlier, boundary),
+                earlier.c.created_on < table.c.created_on,
+                earlier.c.reason.is_not_distinct_from(table.c.reason),
+                sa.not_(reason_changed_between),
+            )
+        )
+        .correlate(table, boundary)
+    )
+    # A ``force`` attempt is an operator action the writer never suppresses:
+    # audit.py's ``_suppress_redundant_block`` only collapses consecutive
+    # scheduled same-reason blocks, so the pruner does not collapse a force row
+    # either — it is never reported as a repeat. This keeps it out of the
+    # duplicate category and (via ``sa.not_`` in the operational category) marks
+    # it a survivor while its streak is current. It is also exempt from
+    # operational age-out (``_operational_candidates`` excludes force blocks), so
+    # a force block is retained permanently — full immortality for operator
+    # force-purge blocks. A force row may still be the *earlier* anchor a later
+    # scheduled repeat collapses into — only the force row itself is protected
+    # from duplicate removal.
+    return sa.and_(table.c.trigger != TRIGGER_FORCE, repeats)
+
+
+def _preceded_by_unresolved_attempt(table: sa.Table) -> sa.ColumnElement[bool]:
+    """Whether an unresolved (``pending``) attempt precedes this row.
+
+    A ``pending`` row is the only thing that can insert a streak boundary
+    into *history*: every other write lands at ``now``, newer than every
+    existing row, whereas reconciliation and the purge path finalize a
+    pending row **in place**, keeping its original ``created_on``. So a
+    pending row sitting between two blocked rows is a boundary that may
+    appear at any moment, and the blocked row after it would become the new
+    streak's survivor — the very row pruning must never delete.
+
+    A tied timestamp counts as preceding. Which of the two writes landed
+    first is unknowable from the row, the block's classification (current
+    duplicate, or resolved-streak row on an age window) changes with the
+    attempt's outcome, and deferring it until then costs nothing.
+    """
+    pending: sa.FromClause = table.alias("unresolved_attempt")
+    return sa.exists(
+        sa.select(sa.literal(1))
+        .select_from(pending)
+        .where(
+            sa.and_(
+                pending.c.status == STATUS_PENDING,
+                pending.c.entity_type == table.c.entity_type,
+                pending.c.entity_uuid == table.c.entity_uuid,
+                pending.c.created_on <= table.c.created_on,
+            )
+        )
+    )
+
+
+def _duplicate_candidates(now: datetime, limit: int) -> sa.sql.Select:
+    """Select current-streak blocked rows that repeat the block before them.
+
+    Age-independent by design (FR-003): a repeat is prunable the moment the
+    streak holds an earlier same-reason block, regardless of the retention
+    window. Trigger does not gate streak membership — ``scheduled`` and
+    ``force`` blocked rows share streaks — but a ``force`` row is itself never
+    collapsed as a repeat (see :func:`_repeats_an_earlier_block`). Reason *is*
+    a discriminator: the first block after a reason change survives alongside
+    the streak's earliest row.
+
+    Rows preceded by an unresolved attempt are skipped: their classification
+    is not stable, because that attempt can finalize into a boundary and
+    promote them to survivor between selection and deletion. Such rows are
+    collected once the attempt resolves. Note that pruning does not depend on
+    that happening promptly — reconciliation runs from the purge task, which
+    a deployment may have disabled or left in dry-run — so a long-lived
+    pending row defers its successors indefinitely rather than risking them.
+    """
+    table: sa.Table = PurgeAuditLog.__table__
+    boundary: sa.Subquery = _streak_boundary_subquery(now)
+    return (
+        sa.select(table.c.id)
+        .select_from(_with_boundary(table, boundary))
+        .where(table.c.status == STATUS_BLOCKED)
+        .where(table.c.entity_uuid.is_not(None))
+        .where(table.c.created_on <= now)
+        .where(_in_current_streak(table, boundary))
+        .where(_repeats_an_earlier_block(table, boundary))
+        .where(sa.not_(_preceded_by_unresolved_attempt(table)))
+        .order_by(table.c.created_on)
+        .limit(limit)

Review Comment:
   Agreed — the 500 cap bounds deletions, not discovery, so the terminal (and, when duplicates are sparse, the first) batch scans O(blocked rows) under the coordination lock. Proposing the fix before I push it, since it changes the lock model you flagged:
   
   **Split discovery from the locked mutation.** Run the candidate SELECT (LIMIT 500) UNLOCKED to get ≤500 candidate ids — a hint, no lock held for the scan. Then take the coordination lock and issue a DELETE scoped to those ids that re-applies the candidacy predicates as correlated WHERE clauses on the id-filtered target (not a re-embedded candidacy sub-SELECT): `DELETE … WHERE id IN (:ids) AND status='blocked' AND <in_current_streak> AND <repeats_an_earlier_block> AND NOT <preceded_by_unresolved_attempt>`. The planner evaluates those correlated, index-backed predicates for only the ≤500 rows, so the locked work is bounded to ~500 PK probes plus their per-entity index lookups.
   
   This keeps the backdated-write invariant intact: the LOCKED delete re-verifies the full predicate over current committed state, so a row a concurrent `write_ahead` turned into a survivor between the unlocked discovery and the lock fails the re-check and is not deleted (a stale hint only ever deletes fewer rows, never wrong ones); a `write_ahead` arriving after the lock waits and stamps post-cutoff. The rowcount-keyed drain loop converges the remainder next run, as today.
   
   I'll attach an `EXPLAIN` of the locked delete showing it's ≤500-bounded (index probes, no seq scan / full per-entity aggregate) — that doubles as the runtime evidence you asked for. Does this shape address your concern? I'll push the diff + EXPLAIN for review rather than merging first.
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org


Re: [PR] feat(deletion-retention): scheduled pruning policy for the purge audit log [superset]

Posted by "mikebridge (via GitHub)" <gi...@apache.org>.
mikebridge commented on code in PR #43490:
URL: https://github.com/apache/superset/pull/43490#discussion_r3906575600


##########
superset/commands/deletion_retention/prune_audit.py:
##########
@@ -0,0 +1,607 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Deletion-only pruning of the ``purge_audit_log`` table.
+
+Bounds the audit history's growth without ever weakening its evidentiary
+value. Three delete categories, applied in priority order under one shared
+per-run batch budget:
+
+1. **Blocked duplicates** — within an entity's *current* blockage streak
+   (its ``blocked`` rows newer than the entity's newest streak-breaking
+   row), only the streak's earliest row survives: later duplicates are
+   removed regardless of age. The survivor carries the "blocked since"
+   fact and is never deleted while the streak is current.
+2. **Operational expiry** — ``blocked`` rows of *resolved* streaks and
+   ``failed`` rows older than ``PURGE_AUDIT_OPERATIONAL_RETENTION_DAYS``
+   age out.
+3. **Evidence expiry** — ``confirmed`` / ``target_absent`` rows are
+   untouchable unless ``PURGE_AUDIT_EVIDENCE_RETENTION_DAYS`` is
+   explicitly set (the operator's compliance assertion), and then only
+   rows older than that window.
+
+``pending`` rows belong to :func:`audit.reconcile_pending` and rows with
+future timestamps (clock skew) are excluded from every category *and* from
+streak classification, so a skewed writer cannot reclassify a live streak.
+
+Three invariants keep the survivor safe without a distributed lock, which
+matters because runs can overlap and both ``reconcile_pending`` and the
+purge path finalize rows concurrently. A duplicate is only ever deleted
+when no concurrent transition could turn it into a survivor first:
+
+* **A boundary is never removed while it still bounds anything.** Evidence
+  expiry refuses to delete a row while an older ``blocked`` or ``pending``
+  row for the same entity survives. Boundaries therefore never *recede*,
+  which would otherwise promote resolved rows into a current streak.
+* **A blocked row whose classification is unstable is never deleted.**
+  Finalizing a ``pending`` row resolves it *in place*, keeping its original
+  timestamp, so an unresolved attempt is a boundary that may appear
+  mid-history at any moment — and the blocked row after it would become its
+  streak's survivor. Blocked rows preceded by an unresolved attempt are
+  therefore skipped by **both** the duplicate and the operational category.
+  Age does not stabilize the classification: an old blocked row inside a
+  live streak is exactly the "blocked for years" case FR-009 protects.
+  Without this, a boundary moving *forward* would demote the current
+  survivor and promote the next row into its place — possibly a row already
+  selected for deletion.
+* **Deletes are conditional and counted from rowcounts.** A row whose
+  status changed since selection is not matched, so overlapping runs can
+  neither double-remove nor double-report.
+
+Boundaries moving forward past *all* of an entity's blocked rows leave no
+survivor to promote, so a selected duplicate may be removed slightly ahead
+of its retention window in that case. It was redundant either way and the
+streak's earliest row is untouched.
+
+Candidate selection is embedded in each ``DELETE`` statement. The derived-table
+wrapper keeps that shape legal on MySQL while ensuring that a pending or
+recovered row committed before the delete is evaluated participates in the
+survivor and boundary predicates. No stale list of candidate ids crosses a
+transaction boundary.
+
+Audit creation/recovery and every pruning batch take the same singleton
+database write lock before assigning a timestamp or evaluating candidates.
+The lock is held through commit, so an audit row cannot become visible in an
+already-processed logical past. Automatic pruning still ships disabled by
+default so operators explicitly choose their retention policy.
+"""
+
+from __future__ import annotations
+
+import logging
+from collections.abc import Callable
+from dataclasses import dataclass, field
+from datetime import datetime, timedelta
+from functools import partial
+from typing import Any, Literal, NamedTuple, TypeAlias
+
+import sqlalchemy as sa
+from flask import current_app
+
+from superset import db
+from superset.commands.deletion_retention.audit import (
+    acquire_coordination_lock,
+    utc_now,
+)
+from superset.models.purge_audit_log import (
+    PurgeAuditLog,
+    STATUS_BLOCKED,
+    STATUS_CONFIRMED,
+    STATUS_FAILED,
+    STATUS_PENDING,
+    STATUS_TARGET_ABSENT,
+)
+
+logger: logging.Logger = logging.getLogger(__name__)
+
+#: Operational records: noise-prone outcomes whose compliance value decays —
+#: a blocked or failed purge leaves the object in place (FR-001).
+OPERATIONAL_STATUSES: frozenset[str] = frozenset({STATUS_BLOCKED, STATUS_FAILED})
+#: Protected evidence: the only surviving trace of a destroyed object.
+PROTECTED_STATUSES: frozenset[str] = frozenset({STATUS_CONFIRMED, STATUS_TARGET_ABSENT})
+#: The outcomes that end a blockage streak — proof the object is gone.
+#:
+#: ``failed`` is deliberately absent. A failed purge is an infrastructure
+#: outcome (the cascade raised), not evidence the blockage cleared: the
+#: policy that blocked the entity is untouched, so the blockage continues
+#: across it. Treating ``failed`` as a boundary would let one transient
+#: error demote the "blocked since" survivor to an ageing duplicate and
+#: restate the blockage as beginning after the failure — losing exactly the
+#: fact FR-003 exists to preserve. ``pending`` is provisional and likewise
+#: neither joins nor breaks streaks.
+_STREAK_BREAKING_STATUSES: frozenset[str] = frozenset(
+    {STATUS_CONFIRMED, STATUS_TARGET_ABSENT}
+)
+
+#: Rows deleted per statement, matching the purge task's batch convention.
+BATCH_SIZE: int = 500
+#: One shared budget for the whole run across all three categories; the
+#: remaining backlog carries over to the next scheduled run (FR-004/SC-004).
+MAX_BATCHES_PER_RUN: int = 10
+
+OPERATIONAL_RETENTION_KEY: str = "PURGE_AUDIT_OPERATIONAL_RETENTION_DAYS"
+EVIDENCE_RETENTION_KEY: str = "PURGE_AUDIT_EVIDENCE_RETENTION_DAYS"
+
+
+class ResolvedWindow(NamedTuple):
+    """A retention window resolved from config.
+
+    Distinguishes the three outcomes a caller must tell apart: a usable
+    window, the deliberate "off" default, and operator error. Collapsing
+    the last two into a bare ``None`` would force every caller to re-read
+    config to find out which it got.
+    """
+
+    days: int | None
+    invalid_key: str | None = None
+
+
+def _validated_window(key: str, value: Any) -> ResolvedWindow:
+    """Validate a configured day count, failing closed on anything odd.
+
+    An invalid value disables its category for the run rather than widening
+    removal (FR-005/SC-005).
+    """
+    # bool is an int subclass and floats would silently truncate — both are
+    # config mistakes, not day counts, on a knob that deletes rows.
+    if not isinstance(value, bool) and not isinstance(value, float):
+        try:
+            days: int = int(value)
+        except (TypeError, ValueError):
+            days = 0
+        if days > 0:
+            return ResolvedWindow(days)
+    logger.warning(
+        "prune_audit: invalid %s=%r; skipping this category for the run "
+        "(pruning never widens on bad configuration)",
+        key,
+        value,
+    )
+    return ResolvedWindow(None, key)
+
+
+def resolve_operational_retention_days() -> ResolvedWindow:
+    """The operational retention window, or a disabled window when invalid."""
+    return _validated_window(
+        OPERATIONAL_RETENTION_KEY, current_app.config.get(OPERATIONAL_RETENTION_KEY)
+    )
+
+
+def resolve_evidence_retention_days() -> ResolvedWindow:
+    """The evidence expiration window; disabled unless explicitly opted in.
+
+    Unset is the documented "never expire evidence" default (FR-006), not an
+    error, so it produces a disabled window with no warning.
+    """
+    value: Any = current_app.config.get(EVIDENCE_RETENTION_KEY)
+    if value is None:
+        return ResolvedWindow(None)
+    return _validated_window(EVIDENCE_RETENTION_KEY, value)
+
+
+@dataclass
+class PruneRunResult:
+    """Per-category removal counts and run disposition for one pruning run."""
+
+    blocked_duplicates: int = 0
+    operational_expired: int = 0
+    evidence_expired: int = 0
+    #: True when the shared batch budget ran out before every category's
+    #: candidates were drained; the remainder converges on later runs.
+    carried_over: bool = False
+    invalid_config_keys: list[str] = field(default_factory=list)
+
+    @property
+    def total_removed(self) -> int:
+        """Total rows removed across every category."""
+        return (
+            self.blocked_duplicates + self.operational_expired + self.evidence_expired
+        )
+
+    def as_dict(self) -> dict[str, Any]:
+        """The task-return / log-line shape of this result."""
+        return {
+            "removed": {
+                "blocked_duplicates": self.blocked_duplicates,
+                "operational_expired": self.operational_expired,
+                "evidence_expired": self.evidence_expired,
+            },
+            "carried_over": self.carried_over,
+            "invalid_config_keys": list(self.invalid_config_keys),
+        }
+
+
+def _streak_boundary_subquery(now: datetime) -> sa.Subquery:
+    """Per entity, the ``created_on`` of its newest streak-breaking row.
+
+    Entities absent from this subquery have never been proven destroyed —
+    all their blocked rows form one current streak. Future-dated rows are
+    excluded so a skewed writer clock cannot push the boundary ahead of a
+    live streak and make its rows look resolved.
+    """
+    table: sa.Table = PurgeAuditLog.__table__
+    return (
+        sa.select(
+            table.c.entity_type.label("entity_type"),
+            table.c.entity_uuid.label("entity_uuid"),
+            sa.func.max(table.c.created_on).label("boundary"),
+        )
+        .where(table.c.status.in_(_STREAK_BREAKING_STATUSES))
+        .where(table.c.entity_uuid.is_not(None))
+        .where(table.c.created_on <= now)
+        .group_by(table.c.entity_type, table.c.entity_uuid)
+        .subquery("streak_boundary")
+    )
+
+
+def _survivor_subquery(now: datetime) -> sa.Subquery:
+    """Per entity, the ``created_on`` of its current streak's earliest row.
+
+    The survivor is the earliest ``blocked`` row strictly newer than the
+    entity's boundary (or its earliest blocked row outright when no boundary
+    exists). Rows sharing that exact timestamp are all treated as survivors —
+    with microsecond precision ties are pathological, and keeping an extra
+    row errs on the preserving side.
+    """
+    table: sa.Table = PurgeAuditLog.__table__
+    boundary: sa.Subquery = _streak_boundary_subquery(now)
+    return (
+        sa.select(
+            table.c.entity_type.label("entity_type"),
+            table.c.entity_uuid.label("entity_uuid"),
+            sa.func.min(table.c.created_on).label("survivor_created_on"),
+        )
+        .select_from(
+            table.outerjoin(
+                boundary,
+                sa.and_(
+                    table.c.entity_type == boundary.c.entity_type,
+                    table.c.entity_uuid == boundary.c.entity_uuid,
+                ),
+            )
+        )
+        .where(table.c.status == STATUS_BLOCKED)
+        .where(table.c.entity_uuid.is_not(None))
+        .where(table.c.created_on <= now)
+        .where(
+            sa.or_(
+                boundary.c.boundary.is_(None),
+                table.c.created_on > boundary.c.boundary,
+            )
+        )
+        .group_by(table.c.entity_type, table.c.entity_uuid)
+        .subquery("streak_survivor")
+    )
+
+
+def _preceded_by_unresolved_attempt(table: sa.Table) -> sa.ColumnElement[bool]:
+    """Whether an unresolved (``pending``) attempt precedes this row.
+
+    A ``pending`` row is the only thing that can insert a streak boundary
+    into *history*: every other write lands at ``now``, newer than every
+    existing row, whereas reconciliation and the purge path finalize a
+    pending row **in place**, keeping its original ``created_on``. So a
+    pending row sitting between two blocked rows is a boundary that may
+    appear at any moment, and the blocked row after it would become the new
+    streak's survivor — the very row pruning must never delete.
+    """
+    pending: sa.Table = table.alias("unresolved_attempt")
+    return sa.exists(
+        sa.select(sa.literal(1))
+        .select_from(pending)
+        .where(
+            sa.and_(
+                pending.c.status == STATUS_PENDING,
+                pending.c.entity_type == table.c.entity_type,
+                pending.c.entity_uuid == table.c.entity_uuid,
+                pending.c.created_on < table.c.created_on,

Review Comment:
   Fixed in 86e86b4078 — good catch. The guard is now `<=`: a `pending` row tied with a block counts as preceding it, so the block is deferred until the attempt resolves.
   
   I also made the tie policy consistent across the other two predicates rather than just this one, since a tie has to land on the same side everywhere or the invariants disagree with each other:
   
   - **Current-streak membership stays strict** (`created_on > boundary`): a block tied with a boundary sits on the boundary's *resolved* side and ages out on the operational window. This is the one place I deliberately did not go the other way — promoting a same-instant block to the survivor of a *new* current streak would make it exempt from age-out forever for an object the boundary proves was destroyed, which is exactly the failure the boundary invariant exists to prevent (and a real re-block after destruction can't happen within one clock tick).
   - **The evidence guard is now `<=` too**, so a boundary can't be evidence-expired while a block tied with it survives. That's needed for the point above to hold; without it the tied block would be left with no boundary at all.
   - **Tied same-reason blocks are all retained** (neither is earlier than the other).
   
   No ordering key added: ids are uuid4, so there's nothing monotonic to use; `created_on` assigned under the coordination lock remains the order, with ties handled as above. The module docstring now has a tie-policy paragraph.
   
   Tests (Postgres + SQLite): `test_a_block_tied_with_an_unresolved_attempt_is_deferred`, `test_evidence_expiry_spares_a_boundary_tied_with_a_blocked_row`, `test_tied_same_reason_blocks_are_all_retained`.



##########
superset/commands/deletion_retention/prune_audit.py:
##########
@@ -0,0 +1,607 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Deletion-only pruning of the ``purge_audit_log`` table.
+
+Bounds the audit history's growth without ever weakening its evidentiary
+value. Three delete categories, applied in priority order under one shared
+per-run batch budget:
+
+1. **Blocked duplicates** — within an entity's *current* blockage streak
+   (its ``blocked`` rows newer than the entity's newest streak-breaking
+   row), only the streak's earliest row survives: later duplicates are
+   removed regardless of age. The survivor carries the "blocked since"
+   fact and is never deleted while the streak is current.
+2. **Operational expiry** — ``blocked`` rows of *resolved* streaks and
+   ``failed`` rows older than ``PURGE_AUDIT_OPERATIONAL_RETENTION_DAYS``
+   age out.
+3. **Evidence expiry** — ``confirmed`` / ``target_absent`` rows are
+   untouchable unless ``PURGE_AUDIT_EVIDENCE_RETENTION_DAYS`` is
+   explicitly set (the operator's compliance assertion), and then only
+   rows older than that window.
+
+``pending`` rows belong to :func:`audit.reconcile_pending` and rows with
+future timestamps (clock skew) are excluded from every category *and* from
+streak classification, so a skewed writer cannot reclassify a live streak.
+
+Three invariants keep the survivor safe without a distributed lock, which
+matters because runs can overlap and both ``reconcile_pending`` and the
+purge path finalize rows concurrently. A duplicate is only ever deleted
+when no concurrent transition could turn it into a survivor first:
+
+* **A boundary is never removed while it still bounds anything.** Evidence
+  expiry refuses to delete a row while an older ``blocked`` or ``pending``
+  row for the same entity survives. Boundaries therefore never *recede*,
+  which would otherwise promote resolved rows into a current streak.
+* **A blocked row whose classification is unstable is never deleted.**
+  Finalizing a ``pending`` row resolves it *in place*, keeping its original
+  timestamp, so an unresolved attempt is a boundary that may appear
+  mid-history at any moment — and the blocked row after it would become its
+  streak's survivor. Blocked rows preceded by an unresolved attempt are
+  therefore skipped by **both** the duplicate and the operational category.
+  Age does not stabilize the classification: an old blocked row inside a
+  live streak is exactly the "blocked for years" case FR-009 protects.
+  Without this, a boundary moving *forward* would demote the current
+  survivor and promote the next row into its place — possibly a row already
+  selected for deletion.
+* **Deletes are conditional and counted from rowcounts.** A row whose
+  status changed since selection is not matched, so overlapping runs can
+  neither double-remove nor double-report.
+
+Boundaries moving forward past *all* of an entity's blocked rows leave no
+survivor to promote, so a selected duplicate may be removed slightly ahead
+of its retention window in that case. It was redundant either way and the
+streak's earliest row is untouched.
+
+Candidate selection is embedded in each ``DELETE`` statement. The derived-table
+wrapper keeps that shape legal on MySQL while ensuring that a pending or
+recovered row committed before the delete is evaluated participates in the
+survivor and boundary predicates. No stale list of candidate ids crosses a
+transaction boundary.
+
+Audit creation/recovery and every pruning batch take the same singleton
+database write lock before assigning a timestamp or evaluating candidates.
+The lock is held through commit, so an audit row cannot become visible in an
+already-processed logical past. Automatic pruning still ships disabled by
+default so operators explicitly choose their retention policy.
+"""
+
+from __future__ import annotations
+
+import logging
+from collections.abc import Callable
+from dataclasses import dataclass, field
+from datetime import datetime, timedelta
+from functools import partial
+from typing import Any, Literal, NamedTuple, TypeAlias
+
+import sqlalchemy as sa
+from flask import current_app
+
+from superset import db
+from superset.commands.deletion_retention.audit import (
+    acquire_coordination_lock,
+    utc_now,
+)
+from superset.models.purge_audit_log import (
+    PurgeAuditLog,
+    STATUS_BLOCKED,
+    STATUS_CONFIRMED,
+    STATUS_FAILED,
+    STATUS_PENDING,
+    STATUS_TARGET_ABSENT,
+)
+
+logger: logging.Logger = logging.getLogger(__name__)
+
+#: Operational records: noise-prone outcomes whose compliance value decays —
+#: a blocked or failed purge leaves the object in place (FR-001).
+OPERATIONAL_STATUSES: frozenset[str] = frozenset({STATUS_BLOCKED, STATUS_FAILED})
+#: Protected evidence: the only surviving trace of a destroyed object.
+PROTECTED_STATUSES: frozenset[str] = frozenset({STATUS_CONFIRMED, STATUS_TARGET_ABSENT})
+#: The outcomes that end a blockage streak — proof the object is gone.
+#:
+#: ``failed`` is deliberately absent. A failed purge is an infrastructure
+#: outcome (the cascade raised), not evidence the blockage cleared: the
+#: policy that blocked the entity is untouched, so the blockage continues
+#: across it. Treating ``failed`` as a boundary would let one transient
+#: error demote the "blocked since" survivor to an ageing duplicate and
+#: restate the blockage as beginning after the failure — losing exactly the
+#: fact FR-003 exists to preserve. ``pending`` is provisional and likewise
+#: neither joins nor breaks streaks.
+_STREAK_BREAKING_STATUSES: frozenset[str] = frozenset(
+    {STATUS_CONFIRMED, STATUS_TARGET_ABSENT}
+)
+
+#: Rows deleted per statement, matching the purge task's batch convention.
+BATCH_SIZE: int = 500
+#: One shared budget for the whole run across all three categories; the
+#: remaining backlog carries over to the next scheduled run (FR-004/SC-004).
+MAX_BATCHES_PER_RUN: int = 10
+
+OPERATIONAL_RETENTION_KEY: str = "PURGE_AUDIT_OPERATIONAL_RETENTION_DAYS"
+EVIDENCE_RETENTION_KEY: str = "PURGE_AUDIT_EVIDENCE_RETENTION_DAYS"
+
+
+class ResolvedWindow(NamedTuple):
+    """A retention window resolved from config.
+
+    Distinguishes the three outcomes a caller must tell apart: a usable
+    window, the deliberate "off" default, and operator error. Collapsing
+    the last two into a bare ``None`` would force every caller to re-read
+    config to find out which it got.
+    """
+
+    days: int | None
+    invalid_key: str | None = None
+
+
+def _validated_window(key: str, value: Any) -> ResolvedWindow:
+    """Validate a configured day count, failing closed on anything odd.
+
+    An invalid value disables its category for the run rather than widening
+    removal (FR-005/SC-005).
+    """
+    # bool is an int subclass and floats would silently truncate — both are
+    # config mistakes, not day counts, on a knob that deletes rows.
+    if not isinstance(value, bool) and not isinstance(value, float):
+        try:
+            days: int = int(value)
+        except (TypeError, ValueError):
+            days = 0
+        if days > 0:
+            return ResolvedWindow(days)
+    logger.warning(
+        "prune_audit: invalid %s=%r; skipping this category for the run "
+        "(pruning never widens on bad configuration)",
+        key,
+        value,
+    )
+    return ResolvedWindow(None, key)
+
+
+def resolve_operational_retention_days() -> ResolvedWindow:
+    """The operational retention window, or a disabled window when invalid."""
+    return _validated_window(
+        OPERATIONAL_RETENTION_KEY, current_app.config.get(OPERATIONAL_RETENTION_KEY)
+    )
+
+
+def resolve_evidence_retention_days() -> ResolvedWindow:
+    """The evidence expiration window; disabled unless explicitly opted in.
+
+    Unset is the documented "never expire evidence" default (FR-006), not an
+    error, so it produces a disabled window with no warning.
+    """
+    value: Any = current_app.config.get(EVIDENCE_RETENTION_KEY)
+    if value is None:
+        return ResolvedWindow(None)
+    return _validated_window(EVIDENCE_RETENTION_KEY, value)
+
+
+@dataclass
+class PruneRunResult:
+    """Per-category removal counts and run disposition for one pruning run."""
+
+    blocked_duplicates: int = 0
+    operational_expired: int = 0
+    evidence_expired: int = 0
+    #: True when the shared batch budget ran out before every category's
+    #: candidates were drained; the remainder converges on later runs.
+    carried_over: bool = False
+    invalid_config_keys: list[str] = field(default_factory=list)
+
+    @property
+    def total_removed(self) -> int:
+        """Total rows removed across every category."""
+        return (
+            self.blocked_duplicates + self.operational_expired + self.evidence_expired
+        )
+
+    def as_dict(self) -> dict[str, Any]:
+        """The task-return / log-line shape of this result."""
+        return {
+            "removed": {
+                "blocked_duplicates": self.blocked_duplicates,
+                "operational_expired": self.operational_expired,
+                "evidence_expired": self.evidence_expired,
+            },
+            "carried_over": self.carried_over,
+            "invalid_config_keys": list(self.invalid_config_keys),
+        }
+
+
+def _streak_boundary_subquery(now: datetime) -> sa.Subquery:
+    """Per entity, the ``created_on`` of its newest streak-breaking row.
+
+    Entities absent from this subquery have never been proven destroyed —
+    all their blocked rows form one current streak. Future-dated rows are
+    excluded so a skewed writer clock cannot push the boundary ahead of a
+    live streak and make its rows look resolved.
+    """
+    table: sa.Table = PurgeAuditLog.__table__
+    return (
+        sa.select(
+            table.c.entity_type.label("entity_type"),
+            table.c.entity_uuid.label("entity_uuid"),
+            sa.func.max(table.c.created_on).label("boundary"),
+        )
+        .where(table.c.status.in_(_STREAK_BREAKING_STATUSES))
+        .where(table.c.entity_uuid.is_not(None))
+        .where(table.c.created_on <= now)
+        .group_by(table.c.entity_type, table.c.entity_uuid)
+        .subquery("streak_boundary")
+    )
+
+
+def _survivor_subquery(now: datetime) -> sa.Subquery:
+    """Per entity, the ``created_on`` of its current streak's earliest row.
+
+    The survivor is the earliest ``blocked`` row strictly newer than the
+    entity's boundary (or its earliest blocked row outright when no boundary
+    exists). Rows sharing that exact timestamp are all treated as survivors —
+    with microsecond precision ties are pathological, and keeping an extra
+    row errs on the preserving side.
+    """
+    table: sa.Table = PurgeAuditLog.__table__
+    boundary: sa.Subquery = _streak_boundary_subquery(now)
+    return (
+        sa.select(
+            table.c.entity_type.label("entity_type"),
+            table.c.entity_uuid.label("entity_uuid"),
+            sa.func.min(table.c.created_on).label("survivor_created_on"),
+        )
+        .select_from(
+            table.outerjoin(
+                boundary,
+                sa.and_(
+                    table.c.entity_type == boundary.c.entity_type,
+                    table.c.entity_uuid == boundary.c.entity_uuid,
+                ),
+            )
+        )
+        .where(table.c.status == STATUS_BLOCKED)
+        .where(table.c.entity_uuid.is_not(None))
+        .where(table.c.created_on <= now)
+        .where(
+            sa.or_(
+                boundary.c.boundary.is_(None),
+                table.c.created_on > boundary.c.boundary,
+            )
+        )
+        .group_by(table.c.entity_type, table.c.entity_uuid)
+        .subquery("streak_survivor")
+    )
+
+
+def _preceded_by_unresolved_attempt(table: sa.Table) -> sa.ColumnElement[bool]:
+    """Whether an unresolved (``pending``) attempt precedes this row.
+
+    A ``pending`` row is the only thing that can insert a streak boundary
+    into *history*: every other write lands at ``now``, newer than every
+    existing row, whereas reconciliation and the purge path finalize a
+    pending row **in place**, keeping its original ``created_on``. So a
+    pending row sitting between two blocked rows is a boundary that may
+    appear at any moment, and the blocked row after it would become the new
+    streak's survivor — the very row pruning must never delete.
+    """
+    pending: sa.Table = table.alias("unresolved_attempt")
+    return sa.exists(
+        sa.select(sa.literal(1))
+        .select_from(pending)
+        .where(
+            sa.and_(
+                pending.c.status == STATUS_PENDING,
+                pending.c.entity_type == table.c.entity_type,
+                pending.c.entity_uuid == table.c.entity_uuid,
+                pending.c.created_on < table.c.created_on,
+            )
+        )
+    )
+
+
+def _duplicate_candidates(now: datetime, limit: int) -> sa.sql.Select:
+    """Select current-streak blocked rows that are not the streak survivor.
+
+    Age-independent by design (FR-003): a duplicate is prunable the moment a
+    streak has a survivor, regardless of the retention window. Trigger is not
+    a discriminator — ``scheduled`` and ``force`` blocked rows share streaks.
+
+    Rows preceded by an unresolved attempt are skipped: their classification
+    is not stable, because that attempt can finalize into a boundary and
+    promote them to survivor between selection and deletion. Such rows are
+    collected once the attempt resolves. Note that pruning does not depend on
+    that happening promptly — reconciliation runs from the purge task, which
+    a deployment may have disabled or left in dry-run — so a long-lived
+    pending row defers its successors indefinitely rather than risking them.
+    """
+    table: sa.Table = PurgeAuditLog.__table__
+    survivor: sa.Subquery = _survivor_subquery(now)
+    return (
+        sa.select(table.c.id)
+        .select_from(
+            table.join(
+                survivor,
+                sa.and_(
+                    table.c.entity_type == survivor.c.entity_type,
+                    table.c.entity_uuid == survivor.c.entity_uuid,
+                ),
+            )
+        )
+        .where(table.c.status == STATUS_BLOCKED)
+        .where(table.c.created_on > survivor.c.survivor_created_on)

Review Comment:
   Agreed, and fixed in 86e86b4078. The duplicate predicate is now the writer's suppression rule applied retroactively: within a current streak, a blocked row is a repeat only if an earlier same-reason block precedes it with no different-reason block in between (`_repeats_an_earlier_block`). So a streak's survivors are its earliest row plus the first row after every reason change — `report_schedule` → `cascade_integrity_failure` keeps both, and a return to an earlier reason keeps that row as well, matching what `finalize_retention_blocked` retains at write time.
   
   Run heads are exempt from age-out the same way the earliest row was (a years-long blockage keeps its reason history); run heads of *resolved* streaks still age out, the boundary being the evidence there.
   
   The reason comparison is NULL-safe (`IS NOT DISTINCT FROM` / `<=>` / `IS`, pinned by a unit test across the three dialects), so pre-feature reason-less rows form a run of their own that the first coded block ends — the same "a reason-less predecessor never matches" rule the writer uses.
   
   Tests: `test_a_reason_change_retains_the_first_block_of_each_run`, `test_a_reason_less_run_ends_at_the_first_coded_block`, `test_reason_transitions_in_a_resolved_streak_still_age_out`. `config.py` / `UPDATING.md` wording updated. Leaving this thread for you to resolve.



##########
superset/initialization/__init__.py:
##########
@@ -1002,11 +1005,14 @@ def _warn_if_retention_beat_missing(self) -> None:
                 "CeleryConfig or add the module to your override.",
                 self._RETENTION_TASK_MODULE,
             )
-        default_flags = self.config.get("DEFAULT_FEATURE_FLAGS", {})
-        configured_flags = self.config.get("FEATURE_FLAGS", {})
-        soft_delete_enabled = bool(
+        default_flags: dict[str, Any] = self.config.get("DEFAULT_FEATURE_FLAGS", {})
+        configured_flags: dict[str, Any] = self.config.get("FEATURE_FLAGS", {})
+        soft_delete_enabled: bool = bool(
             configured_flags.get("SOFT_DELETE", default_flags.get("SOFT_DELETE", False))
         )
+        audit_pruning_enabled: bool = (
+            self.config.get("PURGE_AUDIT_PRUNING_ENABLED", False) is True
+        )

Review Comment:
   Assessed: valid, fixed in 86e86b4078. The task already failed closed on a non-boolean (`skipped_invalid_config`), but the initializer collapsed such a value into "disabled" and said nothing, so a `"true"` typo left the audit log growing with no startup signal. The initializer now warns, naming the key and the offending value, and still treats the switch as not enabled (no schedule diagnostic fires for it).
   
   Test: `test_non_boolean_audit_pruning_switch_warns_and_stays_disabled`, parametrized over `"true"`, `1`, `"False"`, `0`. Resolving per @rusackas.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org


Re: [PR] feat(deletion-retention): scheduled pruning policy for the purge audit log [superset]

Posted by "mikebridge (via GitHub)" <gi...@apache.org>.
mikebridge commented on PR #43490:
URL: https://github.com/apache/superset/pull/43490#issuecomment-5533818994

   Rebased onto latest `master` (the branch was 225 commits behind) and force-pushed to `5acd469b2a`.
   
   What changed in the rebase:
   - **UPDATING.md conflict resolved keeping both entries** — master's `### Archived dataset purge requires impact confirmation` (from #43724) and this PR's `- The purge audit log can now be pruned automatically…` bullet now coexist under `## Next`.
   - **Migration chain re-pointed to keep a single alembic head.** Master advanced `39097d124752` with `8f31c5d726ab` (`index_dataset_dependency_lookups`), which would have left this PR's `a6c21e5b4d93` as a second head off `39097d124752`. `a6c21e5b4d93` is now chained onto `8f31c5d726ab`, so `c7f53d184ea2` is the sole head. The `enforce-single-migration-head` pre-push gate passes.
   - No code changes beyond the conflict resolution and the down-revision re-point; changed-file pre-commit is clean (mypy / ruff / pylint all green).
   
   @rusackas — heads-up that the existing approval predates **both** the earlier delta-review fix (the differing-reason timestamp-tie that could delete a reason-transition run-head, fixed with inclusive tie bounds + a control-run test) **and** this rebase, so GitHub is still showing the old green badge. Could you take a fresh look before merge rather than merging on the stale approval? Thanks.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org


Re: [PR] feat(deletion-retention): scheduled pruning policy for the purge audit log [superset]

Posted by "mikebridge (via GitHub)" <gi...@apache.org>.
mikebridge commented on code in PR #43490:
URL: https://github.com/apache/superset/pull/43490#discussion_r3931387146


##########
superset/commands/deletion_retention/prune_audit.py:
##########
@@ -0,0 +1,689 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Deletion-only pruning of the ``purge_audit_log`` table.
+
+Bounds the audit history's growth without ever weakening its evidentiary
+value. Three delete categories, applied in priority order under one shared
+per-run batch budget:
+
+1. **Blocked duplicates** — within an entity's *current* blockage streak
+   (its ``blocked`` rows newer than the entity's newest streak-breaking
+   row), only the first row of each run of consecutive same-reason rows
+   survives: the streak's earliest row and the first row after every
+   change of block reason. Later same-reason repeats are removed
+   regardless of age. The survivors carry the "blocked since" fact and
+   the reason history — for coded reasons, the same rows the audit writer's
+   own suppression rule retains (reason-less legacy runs are additionally
+   collapsed to their earliest here) — and are never deleted while the
+   streak is current.
+2. **Operational expiry** — ``blocked`` rows of *resolved* streaks and
+   ``failed`` rows older than ``PURGE_AUDIT_OPERATIONAL_RETENTION_DAYS``
+   age out.
+3. **Evidence expiry** — ``confirmed`` / ``target_absent`` rows are
+   untouchable unless ``PURGE_AUDIT_EVIDENCE_RETENTION_DAYS`` is
+   explicitly set (the operator's compliance assertion), and then only
+   rows older than that window.
+
+``pending`` rows belong to :func:`audit.reconcile_pending` and rows with
+future timestamps (clock skew) are excluded from every category *and* from
+streak classification, so a skewed writer cannot reclassify a live streak.
+
+Three invariants keep the survivor safe without a distributed lock, which
+matters because runs can overlap and both ``reconcile_pending`` and the
+purge path finalize rows concurrently. A duplicate is only ever deleted
+when no concurrent transition could turn it into a survivor first:
+
+* **A boundary is never removed while it still bounds anything.** Evidence
+  expiry refuses to delete a row while an older ``blocked`` or ``pending``
+  row for the same entity survives. Boundaries therefore never *recede*,
+  which would otherwise promote resolved rows into a current streak.
+* **A blocked row whose classification is unstable is never deleted.**
+  Finalizing a ``pending`` row resolves it *in place*, keeping its original
+  timestamp, so an unresolved attempt is a boundary that may appear
+  mid-history at any moment — and the blocked row after it would become its
+  streak's survivor. Blocked rows preceded by an unresolved attempt are
+  therefore skipped by **both** the duplicate and the operational category.
+  Age does not stabilize the classification: an old blocked row inside a
+  live streak is exactly the "blocked for years" case FR-009 protects.
+  Without this, a boundary moving *forward* would demote the current
+  survivor and promote the next row into its place — possibly a row already
+  selected for deletion.
+* **Deletes are conditional and counted from rowcounts.** A row whose
+  status changed since selection is not matched, so overlapping runs can
+  neither double-remove nor double-report.
+
+Boundaries moving forward past *all* of an entity's blocked rows leave no
+survivor to promote, so a selected duplicate may be removed slightly ahead
+of its retention window in that case. It was redundant either way and the
+streak's earliest row is untouched.
+
+Timestamps order rows. Ties are possible — legacy second-precision rows,
+two writers within one clock tick — and are resolved on the preserving
+side: a ``pending`` row tied with a blocked row counts as preceding it (the
+block is deferred until the attempt resolves); a blocked row tied with a
+boundary sits on the boundary's *resolved* side (it ages out instead of
+seeding a new current streak, and the boundary is not removed before it);
+and tied same-reason blocked rows are all retained.
+
+Candidate selection is embedded in each ``DELETE`` statement. The derived-table
+wrapper keeps that shape legal on MySQL while ensuring that a pending or
+recovered row committed before the delete is evaluated participates in the
+survivor and boundary predicates. No stale list of candidate ids crosses a
+transaction boundary.
+
+Audit creation/recovery and every pruning batch take the same singleton
+database write lock before assigning a timestamp or evaluating candidates.
+The lock is held through commit, so an audit row cannot become visible in an
+already-processed logical past. Automatic pruning still ships disabled by
+default so operators explicitly choose their retention policy.
+"""
+
+from __future__ import annotations
+
+import logging
+from collections.abc import Callable
+from dataclasses import dataclass, field
+from datetime import datetime, timedelta
+from functools import partial
+from typing import Any, Literal, NamedTuple, TypeAlias
+
+import sqlalchemy as sa
+from flask import current_app
+
+from superset import db
+from superset.commands.deletion_retention.audit import (
+    acquire_coordination_lock,
+    utc_now,
+)
+from superset.models.purge_audit_log import (
+    PurgeAuditLog,
+    STATUS_BLOCKED,
+    STATUS_CONFIRMED,
+    STATUS_FAILED,
+    STATUS_PENDING,
+    STATUS_TARGET_ABSENT,
+)
+
+logger: logging.Logger = logging.getLogger(__name__)
+
+#: Operational records: noise-prone outcomes whose compliance value decays —
+#: a blocked or failed purge leaves the object in place (FR-001).
+OPERATIONAL_STATUSES: frozenset[str] = frozenset({STATUS_BLOCKED, STATUS_FAILED})
+#: Protected evidence: the only surviving trace of a destroyed object.
+PROTECTED_STATUSES: frozenset[str] = frozenset({STATUS_CONFIRMED, STATUS_TARGET_ABSENT})
+#: The outcomes that end a blockage streak — proof the object is gone.
+#:
+#: ``failed`` is deliberately absent. A failed purge is an infrastructure
+#: outcome (the cascade raised), not evidence the blockage cleared: the
+#: policy that blocked the entity is untouched, so the blockage continues
+#: across it. Treating ``failed`` as a boundary would let one transient
+#: error demote the "blocked since" survivor to an ageing duplicate and
+#: restate the blockage as beginning after the failure — losing exactly the
+#: fact FR-003 exists to preserve. ``pending`` is provisional and likewise
+#: neither joins nor breaks streaks.
+_STREAK_BREAKING_STATUSES: frozenset[str] = frozenset(
+    {STATUS_CONFIRMED, STATUS_TARGET_ABSENT}
+)
+
+#: Rows deleted per statement, matching the purge task's batch convention.
+BATCH_SIZE: int = 500
+#: One shared budget for the whole run across all three categories; the
+#: remaining backlog carries over to the next scheduled run (FR-004/SC-004).
+MAX_BATCHES_PER_RUN: int = 10
+
+OPERATIONAL_RETENTION_KEY: str = "PURGE_AUDIT_OPERATIONAL_RETENTION_DAYS"
+EVIDENCE_RETENTION_KEY: str = "PURGE_AUDIT_EVIDENCE_RETENTION_DAYS"
+
+
+class ResolvedWindow(NamedTuple):
+    """A retention window resolved from config.
+
+    Distinguishes the three outcomes a caller must tell apart: a usable
+    window, the deliberate "off" default, and operator error. Collapsing
+    the last two into a bare ``None`` would force every caller to re-read
+    config to find out which it got.
+    """
+
+    days: int | None
+    invalid_key: str | None = None
+
+
+def _validated_window(key: str, value: Any) -> ResolvedWindow:
+    """Validate a configured day count, failing closed on anything odd.
+
+    An invalid value disables its category for the run rather than widening
+    removal (FR-005/SC-005).
+    """
+    # bool is an int subclass and floats would silently truncate — both are
+    # config mistakes, not day counts, on a knob that deletes rows.
+    if not isinstance(value, bool) and not isinstance(value, float):
+        try:
+            days: int = int(value)
+        except (TypeError, ValueError):
+            days = 0
+        if days > 0:
+            return ResolvedWindow(days)
+    logger.warning(
+        "prune_audit: invalid %s=%r; skipping this category for the run "
+        "(pruning never widens on bad configuration)",
+        key,
+        value,
+    )
+    return ResolvedWindow(None, key)
+
+
+def resolve_operational_retention_days() -> ResolvedWindow:
+    """The operational retention window, or a disabled window when invalid."""
+    return _validated_window(
+        OPERATIONAL_RETENTION_KEY, current_app.config.get(OPERATIONAL_RETENTION_KEY)
+    )
+
+
+def resolve_evidence_retention_days() -> ResolvedWindow:
+    """The evidence expiration window; disabled unless explicitly opted in.
+
+    Unset is the documented "never expire evidence" default (FR-006), not an
+    error, so it produces a disabled window with no warning.
+    """
+    value: Any = current_app.config.get(EVIDENCE_RETENTION_KEY)
+    if value is None:
+        return ResolvedWindow(None)
+    return _validated_window(EVIDENCE_RETENTION_KEY, value)
+
+
+@dataclass
+class PruneRunResult:
+    """Per-category removal counts and run disposition for one pruning run."""
+
+    blocked_duplicates: int = 0
+    operational_expired: int = 0
+    evidence_expired: int = 0
+    #: True when the shared batch budget ran out before every category's
+    #: candidates were drained; the remainder converges on later runs.
+    carried_over: bool = False
+    invalid_config_keys: list[str] = field(default_factory=list)
+
+    @property
+    def total_removed(self) -> int:
+        """Total rows removed across every category."""
+        return (
+            self.blocked_duplicates + self.operational_expired + self.evidence_expired
+        )
+
+    def as_dict(self) -> dict[str, Any]:
+        """The task-return / log-line shape of this result."""
+        return {
+            "removed": {
+                "blocked_duplicates": self.blocked_duplicates,
+                "operational_expired": self.operational_expired,
+                "evidence_expired": self.evidence_expired,
+            },
+            "carried_over": self.carried_over,
+            "invalid_config_keys": list(self.invalid_config_keys),
+        }
+
+
+def _streak_boundary_subquery(now: datetime) -> sa.Subquery:
+    """Per entity, the ``created_on`` of its newest streak-breaking row.
+
+    Entities absent from this subquery have never been proven destroyed —
+    all their blocked rows form one current streak. Future-dated rows are
+    excluded so a skewed writer clock cannot push the boundary ahead of a
+    live streak and make its rows look resolved.
+    """
+    table: sa.Table = PurgeAuditLog.__table__
+    return (
+        sa.select(
+            table.c.entity_type.label("entity_type"),
+            table.c.entity_uuid.label("entity_uuid"),
+            sa.func.max(table.c.created_on).label("boundary"),
+        )
+        .where(table.c.status.in_(_STREAK_BREAKING_STATUSES))
+        .where(table.c.entity_uuid.is_not(None))
+        .where(table.c.created_on <= now)
+        .group_by(table.c.entity_type, table.c.entity_uuid)
+        .subquery("streak_boundary")
+    )
+
+
+def _with_boundary(table: sa.Table, boundary: sa.Subquery) -> sa.Join:
+    """Outer-join each row to its entity's streak boundary (NULL if none)."""
+    return table.outerjoin(
+        boundary,
+        sa.and_(
+            table.c.entity_type == boundary.c.entity_type,
+            table.c.entity_uuid == boundary.c.entity_uuid,
+        ),
+    )
+
+
+def _in_current_streak(
+    row: sa.FromClause, boundary: sa.Subquery
+) -> sa.ColumnElement[bool]:
+    """Whether ``row`` is newer than its entity's boundary (or there is none).
+
+    Strictly newer: a row tied with the boundary sits on its resolved side.
+    A boundary proves the object was gone at that instant, and a block that
+    cannot be ordered after the destruction must not seed a new "current"
+    streak — that would mint a survivor exempt from age-out forever for an
+    object that no longer exists.
+    """
+    return sa.or_(boundary.c.boundary.is_(None), row.c.created_on > boundary.c.boundary)
+
+
+def _repeats_an_earlier_block(
+    table: sa.Table, boundary: sa.Subquery
+) -> sa.ColumnElement[bool]:
+    """Whether a same-reason current-streak block precedes this row with no
+    change of reason in between.
+
+    This is the audit writer's suppression rule
+    (:func:`audit.finalize_retention_blocked`) applied retroactively: a
+    block repeating the reason of the block just before it adds nothing,
+    while the first block after a reason change is the only durable record
+    of the new cause and is retained. The comparison is NULL-safe, so
+    consecutive reason-less (pre-feature) rows count as one run, deduped to
+    the run's earliest, and the first coded block ends that run. Note this
+    is *stricter* than the writer for reason-less rows: the writer never
+    suppresses a reason-less block (``_suppress_redundant_block`` bails on a
+    missing code), so here the pruner additionally collapses legacy
+    pre-feature duplicates — the streak's earliest "blocked since" row is
+    still always kept. Tied same-reason rows are not "earlier" than each
+    other, so all of them are kept.
+    """
+    earlier: sa.FromClause = table.alias("earlier_block")
+    between: sa.FromClause = table.alias("reason_change")
+    reason_changed_between: sa.ColumnElement[bool] = sa.exists(
+        sa.select(sa.literal(1))
+        .select_from(between)
+        .where(
+            sa.and_(
+                between.c.status == STATUS_BLOCKED,
+                between.c.entity_type == table.c.entity_type,
+                between.c.entity_uuid == table.c.entity_uuid,
+                # Inclusive bounds: a differing-reason block sharing an
+                # exact timestamp with either endpoint still breaks the run,
+                # so a reason-transition row tied with a neighbour is
+                # preserved as a run head rather than pruned as a repeat
+                # (the same preserving-side tie rule the pending and evidence
+                # guards use). Inclusive bounds only ever add boundaries —
+                # i.e. only ever preserve more, never delete more.
+                between.c.created_on >= earlier.c.created_on,
+                between.c.created_on <= table.c.created_on,
+                between.c.reason.is_distinct_from(table.c.reason),
+            )
+        )
+        .correlate(table, earlier)
+    )
+    return sa.exists(
+        sa.select(sa.literal(1))
+        .select_from(earlier)
+        .where(
+            sa.and_(
+                earlier.c.status == STATUS_BLOCKED,
+                earlier.c.entity_type == table.c.entity_type,
+                earlier.c.entity_uuid == table.c.entity_uuid,
+                _in_current_streak(earlier, boundary),
+                earlier.c.created_on < table.c.created_on,
+                earlier.c.reason.is_not_distinct_from(table.c.reason),
+                sa.not_(reason_changed_between),
+            )
+        )
+        .correlate(table, boundary)
+    )
+
+
+def _preceded_by_unresolved_attempt(table: sa.Table) -> sa.ColumnElement[bool]:
+    """Whether an unresolved (``pending``) attempt precedes this row.
+
+    A ``pending`` row is the only thing that can insert a streak boundary
+    into *history*: every other write lands at ``now``, newer than every
+    existing row, whereas reconciliation and the purge path finalize a
+    pending row **in place**, keeping its original ``created_on``. So a
+    pending row sitting between two blocked rows is a boundary that may
+    appear at any moment, and the blocked row after it would become the new
+    streak's survivor — the very row pruning must never delete.
+
+    A tied timestamp counts as preceding. Which of the two writes landed
+    first is unknowable from the row, the block's classification (current
+    duplicate, or resolved-streak row on an age window) changes with the
+    attempt's outcome, and deferring it until then costs nothing.
+    """
+    pending: sa.FromClause = table.alias("unresolved_attempt")
+    return sa.exists(
+        sa.select(sa.literal(1))
+        .select_from(pending)
+        .where(
+            sa.and_(
+                pending.c.status == STATUS_PENDING,
+                pending.c.entity_type == table.c.entity_type,
+                pending.c.entity_uuid == table.c.entity_uuid,
+                pending.c.created_on <= table.c.created_on,
+            )
+        )
+    )
+
+
+def _duplicate_candidates(now: datetime, limit: int) -> sa.sql.Select:
+    """Select current-streak blocked rows that repeat the block before them.
+
+    Age-independent by design (FR-003): a repeat is prunable the moment the
+    streak holds an earlier same-reason block, regardless of the retention
+    window. Trigger is not a discriminator — ``scheduled`` and ``force``
+    blocked rows share streaks. Reason *is*: the first block after a reason
+    change survives alongside the streak's earliest row (see
+    :func:`_repeats_an_earlier_block`).
+
+    Rows preceded by an unresolved attempt are skipped: their classification
+    is not stable, because that attempt can finalize into a boundary and
+    promote them to survivor between selection and deletion. Such rows are
+    collected once the attempt resolves. Note that pruning does not depend on
+    that happening promptly — reconciliation runs from the purge task, which
+    a deployment may have disabled or left in dry-run — so a long-lived
+    pending row defers its successors indefinitely rather than risking them.
+    """
+    table: sa.Table = PurgeAuditLog.__table__
+    boundary: sa.Subquery = _streak_boundary_subquery(now)
+    return (
+        sa.select(table.c.id)
+        .select_from(_with_boundary(table, boundary))
+        .where(table.c.status == STATUS_BLOCKED)

Review Comment:
   You're right — this one is **real**. Built your scenario (`blocked(scheduled, R) → blocked(force, R)`, same entity) as an integration test against head `e2dddd9da7`: the force block **was pruned** as a duplicate of the earlier scheduled block. `_duplicate_candidates` treated trigger as a non-discriminator, so it collapsed the force row even though the writer retains force outcomes independently.
   
   **Fixed in `ac9bb3bf34`:** `_repeats_an_earlier_block` now excludes force-trigger rows — the pruner mirrors the writer, whose `_suppress_redundant_block` only collapses consecutive *scheduled* same-reason blocks. A force row is never reported as a repeat, so it's kept out of the duplicate category and marked a survivor while its streak is current. `test_force_block_after_scheduled_same_reason_survives` now passes (it failed before the fix). A force row can still be the *earlier* anchor a later scheduled repeat collapses into — `test_force_anchor_still_collapses_a_later_scheduled_repeat` proves the exemption is surgical (only the force row itself is protected). 34 unit + 117 integration tests green.
   
   **One scope boundary I want your explicit call on** (a 4-lens pass surfaced it, not in your original scenario): the exemption protects a force block from duplicate collapse and from current-streak age-out, but **not** once its streak resolves. A force block sitting behind a `confirmed`/`target_absent` boundary still ages on the operational window like any resolved-streak blocked row (probed: it is deleted). That's consistent with the module's taxonomy — blocked rows are operational and decay; only `confirmed`/`target_absent` are durable evidence, and the boundary records the destruction — so I did **not** extend the exemption to operational aging (that would contradict the documented taxonomy and go beyond this finding). I scoped the docstring/comment accordingly and added `test_resolved_streak_force_block_ages_on_operational_window` to pin it. If you'd rather force records be fully immortal (never auto-aged even in a resolved streak), say so and I'll extend the exemption to t
 he operational category — it's a small, safe change, just a policy call I want you to make on an irreversible feature.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org


Re: [PR] feat(deletion-retention): scheduled pruning policy for the purge audit log [superset]

Posted by "bito-code-review[bot] (via GitHub)" <gi...@apache.org>.
bito-code-review[bot] commented on PR #43490:
URL: https://github.com/apache/superset/pull/43490#issuecomment-5502180738

   AI Code Review is in progress (usually takes 3 to 15 minutes unless it's a very large PR).
   
   <details>
   <summary>
   Bito Usage Guide
   </summary>
   
   **Commands**
         
   Type the following command in the pull request comment and save the comment.
         
   - `/review` - Manually triggers an incremental AI Review.
   - `/review full` - Manually triggers a full AI Review.
   
   - `/pause` - Pauses automatic reviews on this pull request.
   - `/resume` - Resumes automatic reviews.
   - `/resolve` - Marks all Bito-posted review comments as resolved.
   - `/abort` - Cancels all in-progress reviews.
   
         
   Refer to the <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/available-commands" target="_blank" rel="noopener noreferrer">documentation</a> for additional commands.
         
   **Configuration**
   
   This repository uses `Superset` You can customize the agent settings <a href="https://alpha.bito.ai/home/ai-agents/code-review-agent" target="_blank" rel="noopener noreferrer">here</a> or contact your Bito workspace admin at evan@preset.io.
         
   **Documentation & Help**
   - <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/getting-started/install-run-using-bito-cloud/create-or-customize-an-agent-instance" target="_blank" rel="noopener noreferrer">Customize agent settings</a>
   - <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/implementing-custom-code-review-rules" target="_blank" rel="noopener noreferrer">Review rules</a>
   - <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent" target="_blank" rel="noopener noreferrer">General documentation</a>
   - <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/faqs" target="_blank" rel="noopener noreferrer">FAQ</a>
   
   </details>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org


Re: [PR] feat(deletion-retention): scheduled pruning policy for the purge audit log [superset]

Posted by "rusackas (via GitHub)" <gi...@apache.org>.
rusackas commented on PR #43490:
URL: https://github.com/apache/superset/pull/43490#issuecomment-5548181865

   @mikebridge ready for re-review/merge?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org


Re: [PR] feat(deletion-retention): scheduled pruning policy for the purge audit log [superset]

Posted by "rusackas (via GitHub)" <gi...@apache.org>.
rusackas commented on code in PR #43490:
URL: https://github.com/apache/superset/pull/43490#discussion_r3963759940


##########
superset/commands/deletion_retention/prune_audit.py:
##########
@@ -0,0 +1,689 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Deletion-only pruning of the ``purge_audit_log`` table.
+
+Bounds the audit history's growth without ever weakening its evidentiary
+value. Three delete categories, applied in priority order under one shared
+per-run batch budget:
+
+1. **Blocked duplicates** — within an entity's *current* blockage streak
+   (its ``blocked`` rows newer than the entity's newest streak-breaking
+   row), only the first row of each run of consecutive same-reason rows
+   survives: the streak's earliest row and the first row after every
+   change of block reason. Later same-reason repeats are removed
+   regardless of age. The survivors carry the "blocked since" fact and
+   the reason history — for coded reasons, the same rows the audit writer's
+   own suppression rule retains (reason-less legacy runs are additionally
+   collapsed to their earliest here) — and are never deleted while the
+   streak is current.
+2. **Operational expiry** — ``blocked`` rows of *resolved* streaks and
+   ``failed`` rows older than ``PURGE_AUDIT_OPERATIONAL_RETENTION_DAYS``
+   age out.
+3. **Evidence expiry** — ``confirmed`` / ``target_absent`` rows are
+   untouchable unless ``PURGE_AUDIT_EVIDENCE_RETENTION_DAYS`` is
+   explicitly set (the operator's compliance assertion), and then only
+   rows older than that window.
+
+``pending`` rows belong to :func:`audit.reconcile_pending` and rows with
+future timestamps (clock skew) are excluded from every category *and* from
+streak classification, so a skewed writer cannot reclassify a live streak.
+
+Three invariants keep the survivor safe without a distributed lock, which
+matters because runs can overlap and both ``reconcile_pending`` and the
+purge path finalize rows concurrently. A duplicate is only ever deleted
+when no concurrent transition could turn it into a survivor first:
+
+* **A boundary is never removed while it still bounds anything.** Evidence
+  expiry refuses to delete a row while an older ``blocked`` or ``pending``
+  row for the same entity survives. Boundaries therefore never *recede*,
+  which would otherwise promote resolved rows into a current streak.
+* **A blocked row whose classification is unstable is never deleted.**
+  Finalizing a ``pending`` row resolves it *in place*, keeping its original
+  timestamp, so an unresolved attempt is a boundary that may appear
+  mid-history at any moment — and the blocked row after it would become its
+  streak's survivor. Blocked rows preceded by an unresolved attempt are
+  therefore skipped by **both** the duplicate and the operational category.
+  Age does not stabilize the classification: an old blocked row inside a
+  live streak is exactly the "blocked for years" case FR-009 protects.
+  Without this, a boundary moving *forward* would demote the current
+  survivor and promote the next row into its place — possibly a row already
+  selected for deletion.
+* **Deletes are conditional and counted from rowcounts.** A row whose
+  status changed since selection is not matched, so overlapping runs can
+  neither double-remove nor double-report.
+
+Boundaries moving forward past *all* of an entity's blocked rows leave no
+survivor to promote, so a selected duplicate may be removed slightly ahead
+of its retention window in that case. It was redundant either way and the
+streak's earliest row is untouched.
+
+Timestamps order rows. Ties are possible — legacy second-precision rows,
+two writers within one clock tick — and are resolved on the preserving
+side: a ``pending`` row tied with a blocked row counts as preceding it (the
+block is deferred until the attempt resolves); a blocked row tied with a
+boundary sits on the boundary's *resolved* side (it ages out instead of
+seeding a new current streak, and the boundary is not removed before it);
+and tied same-reason blocked rows are all retained.
+
+Candidate selection is embedded in each ``DELETE`` statement. The derived-table
+wrapper keeps that shape legal on MySQL while ensuring that a pending or
+recovered row committed before the delete is evaluated participates in the
+survivor and boundary predicates. No stale list of candidate ids crosses a
+transaction boundary.
+
+Audit creation/recovery and every pruning batch take the same singleton
+database write lock before assigning a timestamp or evaluating candidates.
+The lock is held through commit, so an audit row cannot become visible in an
+already-processed logical past. Automatic pruning still ships disabled by
+default so operators explicitly choose their retention policy.
+"""
+
+from __future__ import annotations
+
+import logging
+from collections.abc import Callable
+from dataclasses import dataclass, field
+from datetime import datetime, timedelta
+from functools import partial
+from typing import Any, Literal, NamedTuple, TypeAlias
+
+import sqlalchemy as sa
+from flask import current_app
+
+from superset import db
+from superset.commands.deletion_retention.audit import (
+    acquire_coordination_lock,
+    utc_now,
+)
+from superset.models.purge_audit_log import (
+    PurgeAuditLog,
+    STATUS_BLOCKED,
+    STATUS_CONFIRMED,
+    STATUS_FAILED,
+    STATUS_PENDING,
+    STATUS_TARGET_ABSENT,
+)
+
+logger: logging.Logger = logging.getLogger(__name__)
+
+#: Operational records: noise-prone outcomes whose compliance value decays —
+#: a blocked or failed purge leaves the object in place (FR-001).
+OPERATIONAL_STATUSES: frozenset[str] = frozenset({STATUS_BLOCKED, STATUS_FAILED})
+#: Protected evidence: the only surviving trace of a destroyed object.
+PROTECTED_STATUSES: frozenset[str] = frozenset({STATUS_CONFIRMED, STATUS_TARGET_ABSENT})
+#: The outcomes that end a blockage streak — proof the object is gone.
+#:
+#: ``failed`` is deliberately absent. A failed purge is an infrastructure
+#: outcome (the cascade raised), not evidence the blockage cleared: the
+#: policy that blocked the entity is untouched, so the blockage continues
+#: across it. Treating ``failed`` as a boundary would let one transient
+#: error demote the "blocked since" survivor to an ageing duplicate and
+#: restate the blockage as beginning after the failure — losing exactly the
+#: fact FR-003 exists to preserve. ``pending`` is provisional and likewise
+#: neither joins nor breaks streaks.
+_STREAK_BREAKING_STATUSES: frozenset[str] = frozenset(
+    {STATUS_CONFIRMED, STATUS_TARGET_ABSENT}
+)
+
+#: Rows deleted per statement, matching the purge task's batch convention.
+BATCH_SIZE: int = 500
+#: One shared budget for the whole run across all three categories; the
+#: remaining backlog carries over to the next scheduled run (FR-004/SC-004).
+MAX_BATCHES_PER_RUN: int = 10
+
+OPERATIONAL_RETENTION_KEY: str = "PURGE_AUDIT_OPERATIONAL_RETENTION_DAYS"
+EVIDENCE_RETENTION_KEY: str = "PURGE_AUDIT_EVIDENCE_RETENTION_DAYS"
+
+
+class ResolvedWindow(NamedTuple):
+    """A retention window resolved from config.
+
+    Distinguishes the three outcomes a caller must tell apart: a usable
+    window, the deliberate "off" default, and operator error. Collapsing
+    the last two into a bare ``None`` would force every caller to re-read
+    config to find out which it got.
+    """
+
+    days: int | None
+    invalid_key: str | None = None
+
+
+def _validated_window(key: str, value: Any) -> ResolvedWindow:
+    """Validate a configured day count, failing closed on anything odd.
+
+    An invalid value disables its category for the run rather than widening
+    removal (FR-005/SC-005).
+    """
+    # bool is an int subclass and floats would silently truncate — both are
+    # config mistakes, not day counts, on a knob that deletes rows.
+    if not isinstance(value, bool) and not isinstance(value, float):
+        try:
+            days: int = int(value)
+        except (TypeError, ValueError):
+            days = 0
+        if days > 0:
+            return ResolvedWindow(days)
+    logger.warning(
+        "prune_audit: invalid %s=%r; skipping this category for the run "
+        "(pruning never widens on bad configuration)",
+        key,
+        value,
+    )
+    return ResolvedWindow(None, key)
+
+
+def resolve_operational_retention_days() -> ResolvedWindow:
+    """The operational retention window, or a disabled window when invalid."""
+    return _validated_window(
+        OPERATIONAL_RETENTION_KEY, current_app.config.get(OPERATIONAL_RETENTION_KEY)
+    )
+
+
+def resolve_evidence_retention_days() -> ResolvedWindow:
+    """The evidence expiration window; disabled unless explicitly opted in.
+
+    Unset is the documented "never expire evidence" default (FR-006), not an
+    error, so it produces a disabled window with no warning.
+    """
+    value: Any = current_app.config.get(EVIDENCE_RETENTION_KEY)
+    if value is None:
+        return ResolvedWindow(None)
+    return _validated_window(EVIDENCE_RETENTION_KEY, value)
+
+
+@dataclass
+class PruneRunResult:
+    """Per-category removal counts and run disposition for one pruning run."""
+
+    blocked_duplicates: int = 0
+    operational_expired: int = 0
+    evidence_expired: int = 0
+    #: True when the shared batch budget ran out before every category's
+    #: candidates were drained; the remainder converges on later runs.
+    carried_over: bool = False
+    invalid_config_keys: list[str] = field(default_factory=list)
+
+    @property
+    def total_removed(self) -> int:
+        """Total rows removed across every category."""
+        return (
+            self.blocked_duplicates + self.operational_expired + self.evidence_expired
+        )
+
+    def as_dict(self) -> dict[str, Any]:
+        """The task-return / log-line shape of this result."""
+        return {
+            "removed": {
+                "blocked_duplicates": self.blocked_duplicates,
+                "operational_expired": self.operational_expired,
+                "evidence_expired": self.evidence_expired,
+            },
+            "carried_over": self.carried_over,
+            "invalid_config_keys": list(self.invalid_config_keys),
+        }
+
+
+def _streak_boundary_subquery(now: datetime) -> sa.Subquery:
+    """Per entity, the ``created_on`` of its newest streak-breaking row.
+
+    Entities absent from this subquery have never been proven destroyed —
+    all their blocked rows form one current streak. Future-dated rows are
+    excluded so a skewed writer clock cannot push the boundary ahead of a
+    live streak and make its rows look resolved.
+    """
+    table: sa.Table = PurgeAuditLog.__table__
+    return (
+        sa.select(
+            table.c.entity_type.label("entity_type"),
+            table.c.entity_uuid.label("entity_uuid"),
+            sa.func.max(table.c.created_on).label("boundary"),
+        )
+        .where(table.c.status.in_(_STREAK_BREAKING_STATUSES))
+        .where(table.c.entity_uuid.is_not(None))
+        .where(table.c.created_on <= now)
+        .group_by(table.c.entity_type, table.c.entity_uuid)
+        .subquery("streak_boundary")
+    )
+
+
+def _with_boundary(table: sa.Table, boundary: sa.Subquery) -> sa.Join:
+    """Outer-join each row to its entity's streak boundary (NULL if none)."""
+    return table.outerjoin(
+        boundary,
+        sa.and_(
+            table.c.entity_type == boundary.c.entity_type,
+            table.c.entity_uuid == boundary.c.entity_uuid,
+        ),
+    )
+
+
+def _in_current_streak(
+    row: sa.FromClause, boundary: sa.Subquery
+) -> sa.ColumnElement[bool]:
+    """Whether ``row`` is newer than its entity's boundary (or there is none).
+
+    Strictly newer: a row tied with the boundary sits on its resolved side.
+    A boundary proves the object was gone at that instant, and a block that
+    cannot be ordered after the destruction must not seed a new "current"
+    streak — that would mint a survivor exempt from age-out forever for an
+    object that no longer exists.
+    """
+    return sa.or_(boundary.c.boundary.is_(None), row.c.created_on > boundary.c.boundary)
+
+
+def _repeats_an_earlier_block(
+    table: sa.Table, boundary: sa.Subquery
+) -> sa.ColumnElement[bool]:
+    """Whether a same-reason current-streak block precedes this row with no
+    change of reason in between.
+
+    This is the audit writer's suppression rule
+    (:func:`audit.finalize_retention_blocked`) applied retroactively: a
+    block repeating the reason of the block just before it adds nothing,
+    while the first block after a reason change is the only durable record
+    of the new cause and is retained. The comparison is NULL-safe, so
+    consecutive reason-less (pre-feature) rows count as one run, deduped to
+    the run's earliest, and the first coded block ends that run. Note this
+    is *stricter* than the writer for reason-less rows: the writer never
+    suppresses a reason-less block (``_suppress_redundant_block`` bails on a
+    missing code), so here the pruner additionally collapses legacy
+    pre-feature duplicates — the streak's earliest "blocked since" row is
+    still always kept. Tied same-reason rows are not "earlier" than each
+    other, so all of them are kept.
+    """
+    earlier: sa.FromClause = table.alias("earlier_block")
+    between: sa.FromClause = table.alias("reason_change")
+    reason_changed_between: sa.ColumnElement[bool] = sa.exists(
+        sa.select(sa.literal(1))
+        .select_from(between)
+        .where(
+            sa.and_(
+                between.c.status == STATUS_BLOCKED,
+                between.c.entity_type == table.c.entity_type,
+                between.c.entity_uuid == table.c.entity_uuid,
+                # Inclusive bounds: a differing-reason block sharing an
+                # exact timestamp with either endpoint still breaks the run,
+                # so a reason-transition row tied with a neighbour is
+                # preserved as a run head rather than pruned as a repeat
+                # (the same preserving-side tie rule the pending and evidence
+                # guards use). Inclusive bounds only ever add boundaries —
+                # i.e. only ever preserve more, never delete more.
+                between.c.created_on >= earlier.c.created_on,
+                between.c.created_on <= table.c.created_on,
+                between.c.reason.is_distinct_from(table.c.reason),
+            )
+        )
+        .correlate(table, earlier)
+    )
+    return sa.exists(
+        sa.select(sa.literal(1))
+        .select_from(earlier)
+        .where(
+            sa.and_(
+                earlier.c.status == STATUS_BLOCKED,
+                earlier.c.entity_type == table.c.entity_type,
+                earlier.c.entity_uuid == table.c.entity_uuid,
+                _in_current_streak(earlier, boundary),
+                earlier.c.created_on < table.c.created_on,
+                earlier.c.reason.is_not_distinct_from(table.c.reason),
+                sa.not_(reason_changed_between),
+            )
+        )
+        .correlate(table, boundary)
+    )
+
+
+def _preceded_by_unresolved_attempt(table: sa.Table) -> sa.ColumnElement[bool]:
+    """Whether an unresolved (``pending``) attempt precedes this row.
+
+    A ``pending`` row is the only thing that can insert a streak boundary
+    into *history*: every other write lands at ``now``, newer than every
+    existing row, whereas reconciliation and the purge path finalize a
+    pending row **in place**, keeping its original ``created_on``. So a
+    pending row sitting between two blocked rows is a boundary that may
+    appear at any moment, and the blocked row after it would become the new
+    streak's survivor — the very row pruning must never delete.
+
+    A tied timestamp counts as preceding. Which of the two writes landed
+    first is unknowable from the row, the block's classification (current
+    duplicate, or resolved-streak row on an age window) changes with the
+    attempt's outcome, and deferring it until then costs nothing.
+    """
+    pending: sa.FromClause = table.alias("unresolved_attempt")
+    return sa.exists(
+        sa.select(sa.literal(1))
+        .select_from(pending)
+        .where(
+            sa.and_(
+                pending.c.status == STATUS_PENDING,
+                pending.c.entity_type == table.c.entity_type,
+                pending.c.entity_uuid == table.c.entity_uuid,
+                pending.c.created_on <= table.c.created_on,
+            )
+        )
+    )
+
+
+def _duplicate_candidates(now: datetime, limit: int) -> sa.sql.Select:
+    """Select current-streak blocked rows that repeat the block before them.
+
+    Age-independent by design (FR-003): a repeat is prunable the moment the
+    streak holds an earlier same-reason block, regardless of the retention
+    window. Trigger is not a discriminator — ``scheduled`` and ``force``
+    blocked rows share streaks. Reason *is*: the first block after a reason
+    change survives alongside the streak's earliest row (see
+    :func:`_repeats_an_earlier_block`).
+
+    Rows preceded by an unresolved attempt are skipped: their classification
+    is not stable, because that attempt can finalize into a boundary and
+    promote them to survivor between selection and deletion. Such rows are
+    collected once the attempt resolves. Note that pruning does not depend on
+    that happening promptly — reconciliation runs from the purge task, which
+    a deployment may have disabled or left in dry-run — so a long-lived
+    pending row defers its successors indefinitely rather than risking them.
+    """
+    table: sa.Table = PurgeAuditLog.__table__
+    boundary: sa.Subquery = _streak_boundary_subquery(now)
+    return (
+        sa.select(table.c.id)
+        .select_from(_with_boundary(table, boundary))
+        .where(table.c.status == STATUS_BLOCKED)

Review Comment:
   Let's keep this simple, going with full immortality: extend the exemption so force blocks never age out on the operational window either, not just during the current streak. We can always add a longer-but-not-forever cleanup pass for these in a follow-up PR later if it turns out to matter, no need to design that now.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org


Re: [PR] feat(deletion-retention): scheduled pruning policy for the purge audit log [superset]

Posted by "bito-code-review[bot] (via GitHub)" <gi...@apache.org>.
bito-code-review[bot] commented on PR #43490:
URL: https://github.com/apache/superset/pull/43490#issuecomment-5594375569

   AI Code Review is in progress (usually takes 3 to 15 minutes unless it's a very large PR).
   
   <details>
   <summary>
   Bito Usage Guide
   </summary>
   
   **Commands**
         
   Type the following command in the pull request comment and save the comment.
         
   - `/review` - Manually triggers an incremental AI Review.
   - `/review full` - Manually triggers a full AI Review.
   
   - `/pause` - Pauses automatic reviews on this pull request.
   - `/resume` - Resumes automatic reviews.
   - `/resolve` - Marks all Bito-posted review comments as resolved.
   - `/abort` - Cancels all in-progress reviews.
   
         
   Refer to the <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/available-commands" target="_blank" rel="noopener noreferrer">documentation</a> for additional commands.
         
   **Configuration**
   
   This repository uses `Superset` You can customize the agent settings <a href="https://alpha.bito.ai/home/ai-agents/code-review-agent" target="_blank" rel="noopener noreferrer">here</a> or contact your Bito workspace admin at evan@preset.io.
         
   **Documentation & Help**
   - <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/getting-started/install-run-using-bito-cloud/create-or-customize-an-agent-instance" target="_blank" rel="noopener noreferrer">Customize agent settings</a>
   - <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/implementing-custom-code-review-rules" target="_blank" rel="noopener noreferrer">Review rules</a>
   - <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent" target="_blank" rel="noopener noreferrer">General documentation</a>
   - <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/faqs" target="_blank" rel="noopener noreferrer">FAQ</a>
   
   </details>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org


Re: [PR] feat(deletion-retention): scheduled pruning policy for the purge audit log [superset]

Posted by "bito-code-review[bot] (via GitHub)" <gi...@apache.org>.
bito-code-review[bot] commented on PR #43490:
URL: https://github.com/apache/superset/pull/43490#issuecomment-5626534365

   AI Code Review is in progress (usually takes 3 to 15 minutes unless it's a very large PR).
   
   <details>
   <summary>
   Bito Usage Guide
   </summary>
   
   **Commands**
         
   Type the following command in the pull request comment and save the comment.
         
   - `/review` - Manually triggers an incremental AI Review.
   - `/review full` - Manually triggers a full AI Review.
   
   - `/pause` - Pauses automatic reviews on this pull request.
   - `/resume` - Resumes automatic reviews.
   - `/resolve` - Marks all Bito-posted review comments as resolved.
   - `/abort` - Cancels all in-progress reviews.
   
         
   Refer to the <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/available-commands" target="_blank" rel="noopener noreferrer">documentation</a> for additional commands.
         
   **Configuration**
   
   This repository uses `Superset` You can customize the agent settings <a href="https://alpha.bito.ai/home/ai-agents/code-review-agent" target="_blank" rel="noopener noreferrer">here</a> or contact your Bito workspace admin at evan@preset.io.
         
   **Documentation & Help**
   - <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/getting-started/install-run-using-bito-cloud/create-or-customize-an-agent-instance" target="_blank" rel="noopener noreferrer">Customize agent settings</a>
   - <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/implementing-custom-code-review-rules" target="_blank" rel="noopener noreferrer">Review rules</a>
   - <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent" target="_blank" rel="noopener noreferrer">General documentation</a>
   - <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/faqs" target="_blank" rel="noopener noreferrer">FAQ</a>
   
   </details>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org


Re: [PR] feat(deletion-retention): scheduled pruning policy for the purge audit log [superset]

Posted by "sadpandajoe (via GitHub)" <gi...@apache.org>.
sadpandajoe commented on code in PR #43490:
URL: https://github.com/apache/superset/pull/43490#discussion_r3868650380


##########
superset/commands/deletion_retention/prune_audit.py:
##########
@@ -0,0 +1,607 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Deletion-only pruning of the ``purge_audit_log`` table.
+
+Bounds the audit history's growth without ever weakening its evidentiary
+value. Three delete categories, applied in priority order under one shared
+per-run batch budget:
+
+1. **Blocked duplicates** — within an entity's *current* blockage streak
+   (its ``blocked`` rows newer than the entity's newest streak-breaking
+   row), only the streak's earliest row survives: later duplicates are
+   removed regardless of age. The survivor carries the "blocked since"
+   fact and is never deleted while the streak is current.
+2. **Operational expiry** — ``blocked`` rows of *resolved* streaks and
+   ``failed`` rows older than ``PURGE_AUDIT_OPERATIONAL_RETENTION_DAYS``
+   age out.
+3. **Evidence expiry** — ``confirmed`` / ``target_absent`` rows are
+   untouchable unless ``PURGE_AUDIT_EVIDENCE_RETENTION_DAYS`` is
+   explicitly set (the operator's compliance assertion), and then only
+   rows older than that window.
+
+``pending`` rows belong to :func:`audit.reconcile_pending` and rows with
+future timestamps (clock skew) are excluded from every category *and* from
+streak classification, so a skewed writer cannot reclassify a live streak.
+
+Three invariants keep the survivor safe without a distributed lock, which
+matters because runs can overlap and both ``reconcile_pending`` and the
+purge path finalize rows concurrently. A duplicate is only ever deleted
+when no concurrent transition could turn it into a survivor first:
+
+* **A boundary is never removed while it still bounds anything.** Evidence
+  expiry refuses to delete a row while an older ``blocked`` or ``pending``
+  row for the same entity survives. Boundaries therefore never *recede*,
+  which would otherwise promote resolved rows into a current streak.
+* **A blocked row whose classification is unstable is never deleted.**
+  Finalizing a ``pending`` row resolves it *in place*, keeping its original
+  timestamp, so an unresolved attempt is a boundary that may appear
+  mid-history at any moment — and the blocked row after it would become its
+  streak's survivor. Blocked rows preceded by an unresolved attempt are
+  therefore skipped by **both** the duplicate and the operational category.
+  Age does not stabilize the classification: an old blocked row inside a
+  live streak is exactly the "blocked for years" case FR-009 protects.
+  Without this, a boundary moving *forward* would demote the current
+  survivor and promote the next row into its place — possibly a row already
+  selected for deletion.
+* **Deletes are conditional and counted from rowcounts.** A row whose
+  status changed since selection is not matched, so overlapping runs can
+  neither double-remove nor double-report.
+
+Boundaries moving forward past *all* of an entity's blocked rows leave no
+survivor to promote, so a selected duplicate may be removed slightly ahead
+of its retention window in that case. It was redundant either way and the
+streak's earliest row is untouched.
+
+Candidate selection is embedded in each ``DELETE`` statement. The derived-table
+wrapper keeps that shape legal on MySQL while ensuring that a pending or
+recovered row committed before the delete is evaluated participates in the
+survivor and boundary predicates. No stale list of candidate ids crosses a
+transaction boundary.
+
+Audit creation/recovery and every pruning batch take the same singleton
+database write lock before assigning a timestamp or evaluating candidates.
+The lock is held through commit, so an audit row cannot become visible in an
+already-processed logical past. Automatic pruning still ships disabled by
+default so operators explicitly choose their retention policy.
+"""
+
+from __future__ import annotations
+
+import logging
+from collections.abc import Callable
+from dataclasses import dataclass, field
+from datetime import datetime, timedelta
+from functools import partial
+from typing import Any, Literal, NamedTuple, TypeAlias
+
+import sqlalchemy as sa
+from flask import current_app
+
+from superset import db
+from superset.commands.deletion_retention.audit import (
+    acquire_coordination_lock,
+    utc_now,
+)
+from superset.models.purge_audit_log import (
+    PurgeAuditLog,
+    STATUS_BLOCKED,
+    STATUS_CONFIRMED,
+    STATUS_FAILED,
+    STATUS_PENDING,
+    STATUS_TARGET_ABSENT,
+)
+
+logger: logging.Logger = logging.getLogger(__name__)
+
+#: Operational records: noise-prone outcomes whose compliance value decays —
+#: a blocked or failed purge leaves the object in place (FR-001).
+OPERATIONAL_STATUSES: frozenset[str] = frozenset({STATUS_BLOCKED, STATUS_FAILED})
+#: Protected evidence: the only surviving trace of a destroyed object.
+PROTECTED_STATUSES: frozenset[str] = frozenset({STATUS_CONFIRMED, STATUS_TARGET_ABSENT})
+#: The outcomes that end a blockage streak — proof the object is gone.
+#:
+#: ``failed`` is deliberately absent. A failed purge is an infrastructure
+#: outcome (the cascade raised), not evidence the blockage cleared: the
+#: policy that blocked the entity is untouched, so the blockage continues
+#: across it. Treating ``failed`` as a boundary would let one transient
+#: error demote the "blocked since" survivor to an ageing duplicate and
+#: restate the blockage as beginning after the failure — losing exactly the
+#: fact FR-003 exists to preserve. ``pending`` is provisional and likewise
+#: neither joins nor breaks streaks.
+_STREAK_BREAKING_STATUSES: frozenset[str] = frozenset(
+    {STATUS_CONFIRMED, STATUS_TARGET_ABSENT}
+)
+
+#: Rows deleted per statement, matching the purge task's batch convention.
+BATCH_SIZE: int = 500
+#: One shared budget for the whole run across all three categories; the
+#: remaining backlog carries over to the next scheduled run (FR-004/SC-004).
+MAX_BATCHES_PER_RUN: int = 10
+
+OPERATIONAL_RETENTION_KEY: str = "PURGE_AUDIT_OPERATIONAL_RETENTION_DAYS"
+EVIDENCE_RETENTION_KEY: str = "PURGE_AUDIT_EVIDENCE_RETENTION_DAYS"
+
+
+class ResolvedWindow(NamedTuple):
+    """A retention window resolved from config.
+
+    Distinguishes the three outcomes a caller must tell apart: a usable
+    window, the deliberate "off" default, and operator error. Collapsing
+    the last two into a bare ``None`` would force every caller to re-read
+    config to find out which it got.
+    """
+
+    days: int | None
+    invalid_key: str | None = None
+
+
+def _validated_window(key: str, value: Any) -> ResolvedWindow:
+    """Validate a configured day count, failing closed on anything odd.
+
+    An invalid value disables its category for the run rather than widening
+    removal (FR-005/SC-005).
+    """
+    # bool is an int subclass and floats would silently truncate — both are
+    # config mistakes, not day counts, on a knob that deletes rows.
+    if not isinstance(value, bool) and not isinstance(value, float):
+        try:
+            days: int = int(value)
+        except (TypeError, ValueError):
+            days = 0
+        if days > 0:
+            return ResolvedWindow(days)
+    logger.warning(
+        "prune_audit: invalid %s=%r; skipping this category for the run "
+        "(pruning never widens on bad configuration)",
+        key,
+        value,
+    )
+    return ResolvedWindow(None, key)
+
+
+def resolve_operational_retention_days() -> ResolvedWindow:
+    """The operational retention window, or a disabled window when invalid."""
+    return _validated_window(
+        OPERATIONAL_RETENTION_KEY, current_app.config.get(OPERATIONAL_RETENTION_KEY)
+    )
+
+
+def resolve_evidence_retention_days() -> ResolvedWindow:
+    """The evidence expiration window; disabled unless explicitly opted in.
+
+    Unset is the documented "never expire evidence" default (FR-006), not an
+    error, so it produces a disabled window with no warning.
+    """
+    value: Any = current_app.config.get(EVIDENCE_RETENTION_KEY)
+    if value is None:
+        return ResolvedWindow(None)
+    return _validated_window(EVIDENCE_RETENTION_KEY, value)
+
+
+@dataclass
+class PruneRunResult:
+    """Per-category removal counts and run disposition for one pruning run."""
+
+    blocked_duplicates: int = 0
+    operational_expired: int = 0
+    evidence_expired: int = 0
+    #: True when the shared batch budget ran out before every category's
+    #: candidates were drained; the remainder converges on later runs.
+    carried_over: bool = False
+    invalid_config_keys: list[str] = field(default_factory=list)
+
+    @property
+    def total_removed(self) -> int:
+        """Total rows removed across every category."""
+        return (
+            self.blocked_duplicates + self.operational_expired + self.evidence_expired
+        )
+
+    def as_dict(self) -> dict[str, Any]:
+        """The task-return / log-line shape of this result."""
+        return {
+            "removed": {
+                "blocked_duplicates": self.blocked_duplicates,
+                "operational_expired": self.operational_expired,
+                "evidence_expired": self.evidence_expired,
+            },
+            "carried_over": self.carried_over,
+            "invalid_config_keys": list(self.invalid_config_keys),
+        }
+
+
+def _streak_boundary_subquery(now: datetime) -> sa.Subquery:
+    """Per entity, the ``created_on`` of its newest streak-breaking row.
+
+    Entities absent from this subquery have never been proven destroyed —
+    all their blocked rows form one current streak. Future-dated rows are
+    excluded so a skewed writer clock cannot push the boundary ahead of a
+    live streak and make its rows look resolved.
+    """
+    table: sa.Table = PurgeAuditLog.__table__
+    return (
+        sa.select(
+            table.c.entity_type.label("entity_type"),
+            table.c.entity_uuid.label("entity_uuid"),
+            sa.func.max(table.c.created_on).label("boundary"),
+        )
+        .where(table.c.status.in_(_STREAK_BREAKING_STATUSES))
+        .where(table.c.entity_uuid.is_not(None))
+        .where(table.c.created_on <= now)
+        .group_by(table.c.entity_type, table.c.entity_uuid)
+        .subquery("streak_boundary")
+    )
+
+
+def _survivor_subquery(now: datetime) -> sa.Subquery:
+    """Per entity, the ``created_on`` of its current streak's earliest row.
+
+    The survivor is the earliest ``blocked`` row strictly newer than the
+    entity's boundary (or its earliest blocked row outright when no boundary
+    exists). Rows sharing that exact timestamp are all treated as survivors —
+    with microsecond precision ties are pathological, and keeping an extra
+    row errs on the preserving side.
+    """
+    table: sa.Table = PurgeAuditLog.__table__
+    boundary: sa.Subquery = _streak_boundary_subquery(now)
+    return (
+        sa.select(
+            table.c.entity_type.label("entity_type"),
+            table.c.entity_uuid.label("entity_uuid"),
+            sa.func.min(table.c.created_on).label("survivor_created_on"),
+        )
+        .select_from(
+            table.outerjoin(
+                boundary,
+                sa.and_(
+                    table.c.entity_type == boundary.c.entity_type,
+                    table.c.entity_uuid == boundary.c.entity_uuid,
+                ),
+            )
+        )
+        .where(table.c.status == STATUS_BLOCKED)
+        .where(table.c.entity_uuid.is_not(None))
+        .where(table.c.created_on <= now)
+        .where(
+            sa.or_(
+                boundary.c.boundary.is_(None),
+                table.c.created_on > boundary.c.boundary,
+            )
+        )
+        .group_by(table.c.entity_type, table.c.entity_uuid)
+        .subquery("streak_survivor")
+    )
+
+
+def _preceded_by_unresolved_attempt(table: sa.Table) -> sa.ColumnElement[bool]:
+    """Whether an unresolved (``pending``) attempt precedes this row.
+
+    A ``pending`` row is the only thing that can insert a streak boundary
+    into *history*: every other write lands at ``now``, newer than every
+    existing row, whereas reconciliation and the purge path finalize a
+    pending row **in place**, keeping its original ``created_on``. So a
+    pending row sitting between two blocked rows is a boundary that may
+    appear at any moment, and the blocked row after it would become the new
+    streak's survivor — the very row pruning must never delete.
+    """
+    pending: sa.Table = table.alias("unresolved_attempt")
+    return sa.exists(
+        sa.select(sa.literal(1))
+        .select_from(pending)
+        .where(
+            sa.and_(
+                pending.c.status == STATUS_PENDING,
+                pending.c.entity_type == table.c.entity_type,
+                pending.c.entity_uuid == table.c.entity_uuid,
+                pending.c.created_on < table.c.created_on,
+            )
+        )
+    )
+
+
+def _duplicate_candidates(now: datetime, limit: int) -> sa.sql.Select:
+    """Select current-streak blocked rows that are not the streak survivor.
+
+    Age-independent by design (FR-003): a duplicate is prunable the moment a
+    streak has a survivor, regardless of the retention window. Trigger is not
+    a discriminator — ``scheduled`` and ``force`` blocked rows share streaks.
+
+    Rows preceded by an unresolved attempt are skipped: their classification
+    is not stable, because that attempt can finalize into a boundary and
+    promote them to survivor between selection and deletion. Such rows are
+    collected once the attempt resolves. Note that pruning does not depend on
+    that happening promptly — reconciliation runs from the purge task, which
+    a deployment may have disabled or left in dry-run — so a long-lived
+    pending row defers its successors indefinitely rather than risking them.
+    """
+    table: sa.Table = PurgeAuditLog.__table__
+    survivor: sa.Subquery = _survivor_subquery(now)
+    return (
+        sa.select(table.c.id)
+        .select_from(
+            table.join(
+                survivor,
+                sa.and_(
+                    table.c.entity_type == survivor.c.entity_type,
+                    table.c.entity_uuid == survivor.c.entity_uuid,
+                ),
+            )
+        )
+        .where(table.c.status == STATUS_BLOCKED)
+        .where(table.c.created_on > survivor.c.survivor_created_on)

Review Comment:
   This drops every later `blocked` row in a streak without preserving a reason transition. `audit.py` deliberately retains a new record when the blocking reason changes, so a `report_schedule` block followed by a `cascade_integrity_failure` block loses the only durable evidence of the latter on the next prune. Could the duplicate predicate retain the first block for each reason transition?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org


Re: [PR] feat(deletion-retention): scheduled pruning policy for the purge audit log [superset]

Posted by "sadpandajoe (via GitHub)" <gi...@apache.org>.
sadpandajoe commented on code in PR #43490:
URL: https://github.com/apache/superset/pull/43490#discussion_r3856943849


##########
superset/commands/deletion_retention/prune_audit.py:
##########
@@ -0,0 +1,607 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Deletion-only pruning of the ``purge_audit_log`` table.
+
+Bounds the audit history's growth without ever weakening its evidentiary
+value. Three delete categories, applied in priority order under one shared
+per-run batch budget:
+
+1. **Blocked duplicates** — within an entity's *current* blockage streak
+   (its ``blocked`` rows newer than the entity's newest streak-breaking
+   row), only the streak's earliest row survives: later duplicates are
+   removed regardless of age. The survivor carries the "blocked since"
+   fact and is never deleted while the streak is current.
+2. **Operational expiry** — ``blocked`` rows of *resolved* streaks and
+   ``failed`` rows older than ``PURGE_AUDIT_OPERATIONAL_RETENTION_DAYS``
+   age out.
+3. **Evidence expiry** — ``confirmed`` / ``target_absent`` rows are
+   untouchable unless ``PURGE_AUDIT_EVIDENCE_RETENTION_DAYS`` is
+   explicitly set (the operator's compliance assertion), and then only
+   rows older than that window.
+
+``pending`` rows belong to :func:`audit.reconcile_pending` and rows with
+future timestamps (clock skew) are excluded from every category *and* from
+streak classification, so a skewed writer cannot reclassify a live streak.
+
+Three invariants keep the survivor safe without a distributed lock, which
+matters because runs can overlap and both ``reconcile_pending`` and the
+purge path finalize rows concurrently. A duplicate is only ever deleted
+when no concurrent transition could turn it into a survivor first:
+
+* **A boundary is never removed while it still bounds anything.** Evidence
+  expiry refuses to delete a row while an older ``blocked`` or ``pending``
+  row for the same entity survives. Boundaries therefore never *recede*,
+  which would otherwise promote resolved rows into a current streak.
+* **A blocked row whose classification is unstable is never deleted.**
+  Finalizing a ``pending`` row resolves it *in place*, keeping its original
+  timestamp, so an unresolved attempt is a boundary that may appear
+  mid-history at any moment — and the blocked row after it would become its
+  streak's survivor. Blocked rows preceded by an unresolved attempt are
+  therefore skipped by **both** the duplicate and the operational category.
+  Age does not stabilize the classification: an old blocked row inside a
+  live streak is exactly the "blocked for years" case FR-009 protects.
+  Without this, a boundary moving *forward* would demote the current
+  survivor and promote the next row into its place — possibly a row already
+  selected for deletion.
+* **Deletes are conditional and counted from rowcounts.** A row whose
+  status changed since selection is not matched, so overlapping runs can
+  neither double-remove nor double-report.
+
+Boundaries moving forward past *all* of an entity's blocked rows leave no
+survivor to promote, so a selected duplicate may be removed slightly ahead
+of its retention window in that case. It was redundant either way and the
+streak's earliest row is untouched.
+
+Candidate selection is embedded in each ``DELETE`` statement. The derived-table
+wrapper keeps that shape legal on MySQL while ensuring that a pending or
+recovered row committed before the delete is evaluated participates in the
+survivor and boundary predicates. No stale list of candidate ids crosses a
+transaction boundary.
+
+Audit creation/recovery and every pruning batch take the same singleton
+database write lock before assigning a timestamp or evaluating candidates.
+The lock is held through commit, so an audit row cannot become visible in an
+already-processed logical past. Automatic pruning still ships disabled by
+default so operators explicitly choose their retention policy.
+"""
+
+from __future__ import annotations
+
+import logging
+from collections.abc import Callable
+from dataclasses import dataclass, field
+from datetime import datetime, timedelta
+from functools import partial
+from typing import Any, Literal, NamedTuple, TypeAlias
+
+import sqlalchemy as sa
+from flask import current_app
+
+from superset import db
+from superset.commands.deletion_retention.audit import (
+    acquire_coordination_lock,
+    utc_now,
+)
+from superset.models.purge_audit_log import (
+    PurgeAuditLog,
+    STATUS_BLOCKED,
+    STATUS_CONFIRMED,
+    STATUS_FAILED,
+    STATUS_PENDING,
+    STATUS_TARGET_ABSENT,
+)
+
+logger: logging.Logger = logging.getLogger(__name__)
+
+#: Operational records: noise-prone outcomes whose compliance value decays —
+#: a blocked or failed purge leaves the object in place (FR-001).
+OPERATIONAL_STATUSES: frozenset[str] = frozenset({STATUS_BLOCKED, STATUS_FAILED})
+#: Protected evidence: the only surviving trace of a destroyed object.
+PROTECTED_STATUSES: frozenset[str] = frozenset({STATUS_CONFIRMED, STATUS_TARGET_ABSENT})
+#: The outcomes that end a blockage streak — proof the object is gone.
+#:
+#: ``failed`` is deliberately absent. A failed purge is an infrastructure
+#: outcome (the cascade raised), not evidence the blockage cleared: the
+#: policy that blocked the entity is untouched, so the blockage continues
+#: across it. Treating ``failed`` as a boundary would let one transient
+#: error demote the "blocked since" survivor to an ageing duplicate and
+#: restate the blockage as beginning after the failure — losing exactly the
+#: fact FR-003 exists to preserve. ``pending`` is provisional and likewise
+#: neither joins nor breaks streaks.
+_STREAK_BREAKING_STATUSES: frozenset[str] = frozenset(
+    {STATUS_CONFIRMED, STATUS_TARGET_ABSENT}
+)
+
+#: Rows deleted per statement, matching the purge task's batch convention.
+BATCH_SIZE: int = 500
+#: One shared budget for the whole run across all three categories; the
+#: remaining backlog carries over to the next scheduled run (FR-004/SC-004).
+MAX_BATCHES_PER_RUN: int = 10
+
+OPERATIONAL_RETENTION_KEY: str = "PURGE_AUDIT_OPERATIONAL_RETENTION_DAYS"
+EVIDENCE_RETENTION_KEY: str = "PURGE_AUDIT_EVIDENCE_RETENTION_DAYS"
+
+
+class ResolvedWindow(NamedTuple):
+    """A retention window resolved from config.
+
+    Distinguishes the three outcomes a caller must tell apart: a usable
+    window, the deliberate "off" default, and operator error. Collapsing
+    the last two into a bare ``None`` would force every caller to re-read
+    config to find out which it got.
+    """
+
+    days: int | None
+    invalid_key: str | None = None
+
+
+def _validated_window(key: str, value: Any) -> ResolvedWindow:
+    """Validate a configured day count, failing closed on anything odd.
+
+    An invalid value disables its category for the run rather than widening
+    removal (FR-005/SC-005).
+    """
+    # bool is an int subclass and floats would silently truncate — both are
+    # config mistakes, not day counts, on a knob that deletes rows.
+    if not isinstance(value, bool) and not isinstance(value, float):
+        try:
+            days: int = int(value)
+        except (TypeError, ValueError):
+            days = 0
+        if days > 0:
+            return ResolvedWindow(days)
+    logger.warning(
+        "prune_audit: invalid %s=%r; skipping this category for the run "
+        "(pruning never widens on bad configuration)",
+        key,
+        value,
+    )
+    return ResolvedWindow(None, key)
+
+
+def resolve_operational_retention_days() -> ResolvedWindow:
+    """The operational retention window, or a disabled window when invalid."""
+    return _validated_window(
+        OPERATIONAL_RETENTION_KEY, current_app.config.get(OPERATIONAL_RETENTION_KEY)
+    )
+
+
+def resolve_evidence_retention_days() -> ResolvedWindow:
+    """The evidence expiration window; disabled unless explicitly opted in.
+
+    Unset is the documented "never expire evidence" default (FR-006), not an
+    error, so it produces a disabled window with no warning.
+    """
+    value: Any = current_app.config.get(EVIDENCE_RETENTION_KEY)
+    if value is None:
+        return ResolvedWindow(None)
+    return _validated_window(EVIDENCE_RETENTION_KEY, value)
+
+
+@dataclass
+class PruneRunResult:
+    """Per-category removal counts and run disposition for one pruning run."""
+
+    blocked_duplicates: int = 0
+    operational_expired: int = 0
+    evidence_expired: int = 0
+    #: True when the shared batch budget ran out before every category's
+    #: candidates were drained; the remainder converges on later runs.
+    carried_over: bool = False
+    invalid_config_keys: list[str] = field(default_factory=list)
+
+    @property
+    def total_removed(self) -> int:
+        """Total rows removed across every category."""
+        return (
+            self.blocked_duplicates + self.operational_expired + self.evidence_expired
+        )
+
+    def as_dict(self) -> dict[str, Any]:
+        """The task-return / log-line shape of this result."""
+        return {
+            "removed": {
+                "blocked_duplicates": self.blocked_duplicates,
+                "operational_expired": self.operational_expired,
+                "evidence_expired": self.evidence_expired,
+            },
+            "carried_over": self.carried_over,
+            "invalid_config_keys": list(self.invalid_config_keys),
+        }
+
+
+def _streak_boundary_subquery(now: datetime) -> sa.Subquery:
+    """Per entity, the ``created_on`` of its newest streak-breaking row.
+
+    Entities absent from this subquery have never been proven destroyed —
+    all their blocked rows form one current streak. Future-dated rows are
+    excluded so a skewed writer clock cannot push the boundary ahead of a
+    live streak and make its rows look resolved.
+    """
+    table: sa.Table = PurgeAuditLog.__table__
+    return (
+        sa.select(
+            table.c.entity_type.label("entity_type"),
+            table.c.entity_uuid.label("entity_uuid"),
+            sa.func.max(table.c.created_on).label("boundary"),
+        )
+        .where(table.c.status.in_(_STREAK_BREAKING_STATUSES))
+        .where(table.c.entity_uuid.is_not(None))
+        .where(table.c.created_on <= now)
+        .group_by(table.c.entity_type, table.c.entity_uuid)
+        .subquery("streak_boundary")
+    )
+
+
+def _survivor_subquery(now: datetime) -> sa.Subquery:
+    """Per entity, the ``created_on`` of its current streak's earliest row.
+
+    The survivor is the earliest ``blocked`` row strictly newer than the
+    entity's boundary (or its earliest blocked row outright when no boundary
+    exists). Rows sharing that exact timestamp are all treated as survivors —
+    with microsecond precision ties are pathological, and keeping an extra
+    row errs on the preserving side.
+    """
+    table: sa.Table = PurgeAuditLog.__table__
+    boundary: sa.Subquery = _streak_boundary_subquery(now)
+    return (
+        sa.select(
+            table.c.entity_type.label("entity_type"),
+            table.c.entity_uuid.label("entity_uuid"),
+            sa.func.min(table.c.created_on).label("survivor_created_on"),
+        )
+        .select_from(
+            table.outerjoin(
+                boundary,
+                sa.and_(
+                    table.c.entity_type == boundary.c.entity_type,
+                    table.c.entity_uuid == boundary.c.entity_uuid,
+                ),
+            )
+        )
+        .where(table.c.status == STATUS_BLOCKED)
+        .where(table.c.entity_uuid.is_not(None))
+        .where(table.c.created_on <= now)
+        .where(
+            sa.or_(
+                boundary.c.boundary.is_(None),
+                table.c.created_on > boundary.c.boundary,
+            )
+        )
+        .group_by(table.c.entity_type, table.c.entity_uuid)
+        .subquery("streak_survivor")
+    )
+
+
+def _preceded_by_unresolved_attempt(table: sa.Table) -> sa.ColumnElement[bool]:
+    """Whether an unresolved (``pending``) attempt precedes this row.
+
+    A ``pending`` row is the only thing that can insert a streak boundary
+    into *history*: every other write lands at ``now``, newer than every
+    existing row, whereas reconciliation and the purge path finalize a
+    pending row **in place**, keeping its original ``created_on``. So a
+    pending row sitting between two blocked rows is a boundary that may
+    appear at any moment, and the blocked row after it would become the new
+    streak's survivor — the very row pruning must never delete.
+    """
+    pending: sa.Table = table.alias("unresolved_attempt")
+    return sa.exists(
+        sa.select(sa.literal(1))
+        .select_from(pending)
+        .where(
+            sa.and_(
+                pending.c.status == STATUS_PENDING,
+                pending.c.entity_type == table.c.entity_type,
+                pending.c.entity_uuid == table.c.entity_uuid,
+                pending.c.created_on < table.c.created_on,

Review Comment:
   A pending row at the same timestamp is not considered to precede this block. Existing MySQL audit rows can share a timestamp, so `blocked(T0) → pending(T1) → blocked(T1)` lets the later block be pruned; if the pending row later finalizes, that block should become the streak survivor. Could this preserve equal-timestamp successors (or add a deterministic ordering key)?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org


Re: [PR] feat(deletion-retention): scheduled pruning policy for the purge audit log [superset]

Posted by "mikebridge (via GitHub)" <gi...@apache.org>.
mikebridge commented on code in PR #43490:
URL: https://github.com/apache/superset/pull/43490#discussion_r4008044043


##########
superset/commands/deletion_retention/prune_audit.py:
##########
@@ -0,0 +1,726 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Deletion-only pruning of the ``purge_audit_log`` table.
+
+Bounds the audit history's growth without ever weakening its evidentiary
+value. Three delete categories, applied in priority order under one shared
+per-run batch budget:
+
+1. **Blocked duplicates** — within an entity's *current* blockage streak
+   (its ``blocked`` rows newer than the entity's newest streak-breaking
+   row), only the first row of each run of consecutive same-reason rows
+   survives: the streak's earliest row and the first row after every
+   change of block reason. Later same-reason repeats are removed
+   regardless of age. The survivors carry the "blocked since" fact and
+   the reason history — for coded reasons, the same rows the audit writer's
+   own suppression rule retains (reason-less legacy runs are additionally
+   collapsed to their earliest here) — and are never deleted while the
+   streak is current.
+2. **Operational expiry** — ``blocked`` rows of *resolved* streaks and
+   ``failed`` rows older than ``PURGE_AUDIT_OPERATIONAL_RETENTION_DAYS``
+   age out.
+3. **Evidence expiry** — ``confirmed`` / ``target_absent`` rows are
+   untouchable unless ``PURGE_AUDIT_EVIDENCE_RETENTION_DAYS`` is
+   explicitly set (the operator's compliance assertion), and then only
+   rows older than that window.
+
+``pending`` rows belong to :func:`audit.reconcile_pending` and rows with
+future timestamps (clock skew) are excluded from every category *and* from
+streak classification, so a skewed writer cannot reclassify a live streak.
+
+Three invariants keep the survivor safe without a distributed lock, which
+matters because runs can overlap and both ``reconcile_pending`` and the
+purge path finalize rows concurrently. A duplicate is only ever deleted
+when no concurrent transition could turn it into a survivor first:
+
+* **A boundary is never removed while it still bounds anything.** Evidence
+  expiry refuses to delete a row while an older ``blocked`` or ``pending``
+  row for the same entity survives. Boundaries therefore never *recede*,
+  which would otherwise promote resolved rows into a current streak.
+* **A blocked row whose classification is unstable is never deleted.**
+  Finalizing a ``pending`` row resolves it *in place*, keeping its original
+  timestamp, so an unresolved attempt is a boundary that may appear
+  mid-history at any moment — and the blocked row after it would become its
+  streak's survivor. Blocked rows preceded by an unresolved attempt are
+  therefore skipped by **both** the duplicate and the operational category.
+  Age does not stabilize the classification: an old blocked row inside a
+  live streak is exactly the "blocked for years" case FR-009 protects.
+  Without this, a boundary moving *forward* would demote the current
+  survivor and promote the next row into its place — possibly a row already
+  selected for deletion.
+* **Deletes are conditional and counted from rowcounts.** A row whose
+  status changed since selection is not matched, so overlapping runs can
+  neither double-remove nor double-report.
+
+Boundaries moving forward past *all* of an entity's blocked rows leave no
+survivor to promote, so a selected duplicate may be removed slightly ahead
+of its retention window in that case. It was redundant either way and the
+streak's earliest row is untouched.
+
+Timestamps order rows. Ties are possible — legacy second-precision rows,
+two writers within one clock tick — and are resolved on the preserving
+side: a ``pending`` row tied with a blocked row counts as preceding it (the
+block is deferred until the attempt resolves); a blocked row tied with a
+boundary sits on the boundary's *resolved* side (it ages out instead of
+seeding a new current streak, and the boundary is not removed before it);
+and tied same-reason blocked rows are all retained.
+
+Candidate selection is embedded in each ``DELETE`` statement. The derived-table
+wrapper keeps that shape legal on MySQL while ensuring that a pending or
+recovered row committed before the delete is evaluated participates in the
+survivor and boundary predicates. No stale list of candidate ids crosses a
+transaction boundary.
+
+Audit creation/recovery and every pruning batch take the same singleton
+database write lock before assigning a timestamp or evaluating candidates.
+The lock is held through commit, so an audit row cannot become visible in an
+already-processed logical past. Automatic pruning still ships disabled by
+default so operators explicitly choose their retention policy.
+"""
+
+from __future__ import annotations
+
+import logging
+from collections.abc import Callable
+from dataclasses import dataclass, field
+from datetime import datetime, timedelta
+from functools import partial
+from typing import Any, Literal, NamedTuple, TypeAlias
+
+import sqlalchemy as sa
+from flask import current_app
+
+from superset import db
+from superset.commands.deletion_retention.audit import (
+    acquire_coordination_lock,
+    TRIGGER_FORCE,
+    utc_now,
+)
+from superset.models.purge_audit_log import (
+    PurgeAuditLog,
+    STATUS_BLOCKED,
+    STATUS_CONFIRMED,
+    STATUS_FAILED,
+    STATUS_PENDING,
+    STATUS_TARGET_ABSENT,
+)
+
+logger: logging.Logger = logging.getLogger(__name__)
+
+#: Operational records: noise-prone outcomes whose compliance value decays —
+#: a blocked or failed purge leaves the object in place (FR-001).
+OPERATIONAL_STATUSES: frozenset[str] = frozenset({STATUS_BLOCKED, STATUS_FAILED})
+#: Protected evidence: the only surviving trace of a destroyed object.
+PROTECTED_STATUSES: frozenset[str] = frozenset({STATUS_CONFIRMED, STATUS_TARGET_ABSENT})
+#: The outcomes that end a blockage streak — proof the object is gone.
+#:
+#: ``failed`` is deliberately absent. A failed purge is an infrastructure
+#: outcome (the cascade raised), not evidence the blockage cleared: the
+#: policy that blocked the entity is untouched, so the blockage continues
+#: across it. Treating ``failed`` as a boundary would let one transient
+#: error demote the "blocked since" survivor to an ageing duplicate and
+#: restate the blockage as beginning after the failure — losing exactly the
+#: fact FR-003 exists to preserve. ``pending`` is provisional and likewise
+#: neither joins nor breaks streaks.
+_STREAK_BREAKING_STATUSES: frozenset[str] = frozenset(
+    {STATUS_CONFIRMED, STATUS_TARGET_ABSENT}
+)
+
+#: Rows deleted per statement, matching the purge task's batch convention.
+BATCH_SIZE: int = 500
+#: One shared budget for the whole run across all three categories; the
+#: remaining backlog carries over to the next scheduled run (FR-004/SC-004).
+MAX_BATCHES_PER_RUN: int = 10
+
+OPERATIONAL_RETENTION_KEY: str = "PURGE_AUDIT_OPERATIONAL_RETENTION_DAYS"
+EVIDENCE_RETENTION_KEY: str = "PURGE_AUDIT_EVIDENCE_RETENTION_DAYS"
+
+
+class ResolvedWindow(NamedTuple):
+    """A retention window resolved from config.
+
+    Distinguishes the three outcomes a caller must tell apart: a usable
+    window, the deliberate "off" default, and operator error. Collapsing
+    the last two into a bare ``None`` would force every caller to re-read
+    config to find out which it got.
+    """
+
+    days: int | None
+    invalid_key: str | None = None
+
+
+def _validated_window(key: str, value: Any) -> ResolvedWindow:
+    """Validate a configured day count, failing closed on anything odd.
+
+    An invalid value disables its category for the run rather than widening
+    removal (FR-005/SC-005).
+    """
+    # bool is an int subclass and floats would silently truncate — both are
+    # config mistakes, not day counts, on a knob that deletes rows.
+    if not isinstance(value, bool) and not isinstance(value, float):
+        try:
+            days: int = int(value)
+        except (TypeError, ValueError):
+            days = 0
+        if days > 0:
+            return ResolvedWindow(days)
+    logger.warning(
+        "prune_audit: invalid %s=%r; skipping this category for the run "
+        "(pruning never widens on bad configuration)",
+        key,
+        value,
+    )
+    return ResolvedWindow(None, key)
+
+
+def resolve_operational_retention_days() -> ResolvedWindow:
+    """The operational retention window, or a disabled window when invalid."""
+    return _validated_window(
+        OPERATIONAL_RETENTION_KEY, current_app.config.get(OPERATIONAL_RETENTION_KEY)
+    )
+
+
+def resolve_evidence_retention_days() -> ResolvedWindow:
+    """The evidence expiration window; disabled unless explicitly opted in.
+
+    Unset is the documented "never expire evidence" default (FR-006), not an
+    error, so it produces a disabled window with no warning.
+    """
+    value: Any = current_app.config.get(EVIDENCE_RETENTION_KEY)
+    if value is None:
+        return ResolvedWindow(None)
+    return _validated_window(EVIDENCE_RETENTION_KEY, value)
+
+
+@dataclass
+class PruneRunResult:
+    """Per-category removal counts and run disposition for one pruning run."""
+
+    blocked_duplicates: int = 0
+    operational_expired: int = 0
+    evidence_expired: int = 0
+    #: True when the shared batch budget ran out before every category's
+    #: candidates were drained; the remainder converges on later runs.
+    carried_over: bool = False
+    invalid_config_keys: list[str] = field(default_factory=list)
+
+    @property
+    def total_removed(self) -> int:
+        """Total rows removed across every category."""
+        return (
+            self.blocked_duplicates + self.operational_expired + self.evidence_expired
+        )
+
+    def as_dict(self) -> dict[str, Any]:
+        """The task-return / log-line shape of this result."""
+        return {
+            "removed": {
+                "blocked_duplicates": self.blocked_duplicates,
+                "operational_expired": self.operational_expired,
+                "evidence_expired": self.evidence_expired,
+            },
+            "carried_over": self.carried_over,
+            "invalid_config_keys": list(self.invalid_config_keys),
+        }
+
+
+def _streak_boundary_subquery(now: datetime) -> sa.Subquery:
+    """Per entity, the ``created_on`` of its newest streak-breaking row.
+
+    Entities absent from this subquery have never been proven destroyed —
+    all their blocked rows form one current streak. Future-dated rows are
+    excluded so a skewed writer clock cannot push the boundary ahead of a
+    live streak and make its rows look resolved.
+    """
+    table: sa.Table = PurgeAuditLog.__table__
+    return (
+        sa.select(
+            table.c.entity_type.label("entity_type"),
+            table.c.entity_uuid.label("entity_uuid"),
+            sa.func.max(table.c.created_on).label("boundary"),
+        )
+        .where(table.c.status.in_(_STREAK_BREAKING_STATUSES))
+        .where(table.c.entity_uuid.is_not(None))
+        .where(table.c.created_on <= now)
+        .group_by(table.c.entity_type, table.c.entity_uuid)
+        .subquery("streak_boundary")
+    )
+
+
+def _with_boundary(table: sa.Table, boundary: sa.Subquery) -> sa.Join:
+    """Outer-join each row to its entity's streak boundary (NULL if none)."""
+    return table.outerjoin(
+        boundary,
+        sa.and_(
+            table.c.entity_type == boundary.c.entity_type,
+            table.c.entity_uuid == boundary.c.entity_uuid,
+        ),
+    )
+
+
+def _in_current_streak(
+    row: sa.FromClause, boundary: sa.Subquery
+) -> sa.ColumnElement[bool]:
+    """Whether ``row`` is newer than its entity's boundary (or there is none).
+
+    Strictly newer: a row tied with the boundary sits on its resolved side.
+    A boundary proves the object was gone at that instant, and a block that
+    cannot be ordered after the destruction must not seed a new "current"
+    streak — that would mint a survivor exempt from age-out forever for an
+    object that no longer exists.
+    """
+    return sa.or_(boundary.c.boundary.is_(None), row.c.created_on > boundary.c.boundary)
+
+
+def _repeats_an_earlier_block(
+    table: sa.Table, boundary: sa.Subquery
+) -> sa.ColumnElement[bool]:
+    """Whether a same-reason current-streak block precedes this row with no
+    change of reason in between.
+
+    This is the audit writer's suppression rule
+    (:func:`audit.finalize_retention_blocked`) applied retroactively: a
+    block repeating the reason of the block just before it adds nothing,
+    while the first block after a reason change is the only durable record
+    of the new cause and is retained. The comparison is NULL-safe, so
+    consecutive reason-less (pre-feature) rows count as one run, deduped to
+    the run's earliest, and the first coded block ends that run. Note this
+    is *stricter* than the writer for reason-less rows: the writer never
+    suppresses a reason-less block (``_suppress_redundant_block`` bails on a
+    missing code), so here the pruner additionally collapses legacy
+    pre-feature duplicates — the streak's earliest "blocked since" row is
+    still always kept. Tied same-reason rows are not "earlier" than each
+    other, so all of them are kept.
+
+    A ``force`` row is exempt: it is never reported as a repeat (see the
+    return), so it is kept out of the duplicate category and marked a
+    survivor while its streak is current. It is also exempt from operational
+    age-out (see :func:`_operational_candidates`), so an operator force-purge
+    block is retained permanently — never pruned by either category.
+    """
+    earlier: sa.FromClause = table.alias("earlier_block")
+    between: sa.FromClause = table.alias("reason_change")
+    reason_changed_between: sa.ColumnElement[bool] = sa.exists(
+        sa.select(sa.literal(1))
+        .select_from(between)
+        .where(
+            sa.and_(
+                between.c.status == STATUS_BLOCKED,
+                between.c.entity_type == table.c.entity_type,
+                between.c.entity_uuid == table.c.entity_uuid,
+                # Inclusive bounds: a differing-reason block sharing an
+                # exact timestamp with either endpoint still breaks the run,
+                # so a reason-transition row tied with a neighbour is
+                # preserved as a run head rather than pruned as a repeat
+                # (the same preserving-side tie rule the pending and evidence
+                # guards use). Inclusive bounds only ever add boundaries —
+                # i.e. only ever preserve more, never delete more.
+                between.c.created_on >= earlier.c.created_on,
+                between.c.created_on <= table.c.created_on,
+                between.c.reason.is_distinct_from(table.c.reason),
+            )
+        )
+        .correlate(table, earlier)
+    )
+    repeats: sa.ColumnElement[bool] = sa.exists(
+        sa.select(sa.literal(1))
+        .select_from(earlier)
+        .where(
+            sa.and_(
+                earlier.c.status == STATUS_BLOCKED,
+                earlier.c.entity_type == table.c.entity_type,
+                earlier.c.entity_uuid == table.c.entity_uuid,
+                _in_current_streak(earlier, boundary),
+                earlier.c.created_on < table.c.created_on,
+                earlier.c.reason.is_not_distinct_from(table.c.reason),
+                sa.not_(reason_changed_between),
+            )
+        )
+        .correlate(table, boundary)
+    )
+    # A ``force`` attempt is an operator action the writer never suppresses:
+    # audit.py's ``_suppress_redundant_block`` only collapses consecutive
+    # scheduled same-reason blocks, so the pruner does not collapse a force row
+    # either — it is never reported as a repeat. This keeps it out of the
+    # duplicate category and (via ``sa.not_`` in the operational category) marks
+    # it a survivor while its streak is current. It is also exempt from
+    # operational age-out (``_operational_candidates`` excludes force blocks), so
+    # a force block is retained permanently — full immortality for operator
+    # force-purge blocks. A force row may still be the *earlier* anchor a later
+    # scheduled repeat collapses into — only the force row itself is protected
+    # from duplicate removal.
+    return sa.and_(table.c.trigger != TRIGGER_FORCE, repeats)
+
+
+def _preceded_by_unresolved_attempt(table: sa.Table) -> sa.ColumnElement[bool]:
+    """Whether an unresolved (``pending``) attempt precedes this row.
+
+    A ``pending`` row is the only thing that can insert a streak boundary
+    into *history*: every other write lands at ``now``, newer than every
+    existing row, whereas reconciliation and the purge path finalize a
+    pending row **in place**, keeping its original ``created_on``. So a
+    pending row sitting between two blocked rows is a boundary that may
+    appear at any moment, and the blocked row after it would become the new
+    streak's survivor — the very row pruning must never delete.
+
+    A tied timestamp counts as preceding. Which of the two writes landed
+    first is unknowable from the row, the block's classification (current
+    duplicate, or resolved-streak row on an age window) changes with the
+    attempt's outcome, and deferring it until then costs nothing.
+    """
+    pending: sa.FromClause = table.alias("unresolved_attempt")
+    return sa.exists(
+        sa.select(sa.literal(1))
+        .select_from(pending)
+        .where(
+            sa.and_(
+                pending.c.status == STATUS_PENDING,
+                pending.c.entity_type == table.c.entity_type,
+                pending.c.entity_uuid == table.c.entity_uuid,
+                pending.c.created_on <= table.c.created_on,
+            )
+        )
+    )
+
+
+def _duplicate_candidates(now: datetime, limit: int) -> sa.sql.Select:
+    """Select current-streak blocked rows that repeat the block before them.
+
+    Age-independent by design (FR-003): a repeat is prunable the moment the
+    streak holds an earlier same-reason block, regardless of the retention
+    window. Trigger does not gate streak membership — ``scheduled`` and
+    ``force`` blocked rows share streaks — but a ``force`` row is itself never
+    collapsed as a repeat (see :func:`_repeats_an_earlier_block`). Reason *is*
+    a discriminator: the first block after a reason change survives alongside
+    the streak's earliest row.
+
+    Rows preceded by an unresolved attempt are skipped: their classification
+    is not stable, because that attempt can finalize into a boundary and
+    promote them to survivor between selection and deletion. Such rows are
+    collected once the attempt resolves. Note that pruning does not depend on
+    that happening promptly — reconciliation runs from the purge task, which
+    a deployment may have disabled or left in dry-run — so a long-lived
+    pending row defers its successors indefinitely rather than risking them.
+    """
+    table: sa.Table = PurgeAuditLog.__table__
+    boundary: sa.Subquery = _streak_boundary_subquery(now)
+    return (
+        sa.select(table.c.id)
+        .select_from(_with_boundary(table, boundary))
+        .where(table.c.status == STATUS_BLOCKED)
+        .where(table.c.entity_uuid.is_not(None))
+        .where(table.c.created_on <= now)
+        .where(_in_current_streak(table, boundary))
+        .where(_repeats_an_earlier_block(table, boundary))
+        .where(sa.not_(_preceded_by_unresolved_attempt(table)))
+        .order_by(table.c.created_on)
+        .limit(limit)

Review Comment:
   You were right, and I measured it rather than argue it. Setup: one entity with a 6,000-row `blocked` history (5 resolved streaks + a current one, three reasons cycling in runs of 10), the real discovery → real coordination lock → real re-check, with a concurrent writer timing its own `acquire_coordination_lock`; statistics refreshed. Per 500-id batch the lock is held **~6.4 s on PostgreSQL 16** (measured acquire-to-release, median of 5) and **~50 s on MySQL 8 under REPEATABLE READ** (that one is the re-check's `EXPLAIN ANALYZE` actual time — my harness crashed before that run's acquire-to-release loop; the 50- and 100-id MySQL figures below are measured samples). REPEATABLE READ is production's effective level — the READ COMMITTED pinning in `set_db_default_isolation` is a no-op — and the writer waits the whole time. The plans below are exactly your diagnosis: `earlier_block` is probed per candidate with `reason` as a heap filter (4,875 rows removed per loop) and the nes
 ted `reason_change` EXISTS runs 811,370 times — O(history²) per candidate, under the lock.
   
   Two things that did not work, so you don't have to ask:
   1. Adding `reason` to the pruning index: no material change (5.5 s vs 6.4 s). `IS NOT DISTINCT FROM` is not btree-seekable, so the comparison stays a filter even index-only, and the cost is the nested-EXISTS count, not heap fetches.
   2. Rewriting the predicate around the immediately-preceding block: provably equivalent (all 34 behavioural tests unchanged, candidate sets row-for-row identical on the seed, a 40-seed randomised equivalence test) and ~300× faster on Postgres — but MySQL re-evaluates each reference to the correlated scalar with a full per-entity scan, and the locked re-check went from ~50 s to >10 min. Parked, not shipped.
   
   What this PR does: `PURGE_AUDIT_PRUNING_BATCH_SIZE`, default **50** (a non-boolean integer in [1, 500], fail-closed — anything else present skips the run and reports the key). It caps candidates per batch, not lock time — the locked predicate still walks each candidate's history — but on the same seed a batch of 50 holds the lock **~0.15 s on Postgres / ~1.2 s on MySQL**, versus ~0.9 s / ~7.7 s at 100 and ~6.4 s / ~50 s at 500, with the workload-dependent trade-off (more, shorter lock acquisitions; proportionally slower drain per run) documented in `config.py` and UPDATING. The rollback recipe also now requires waiting for every active pruning run to exit, not just running batches, before a writer is rolled back. The real fix is computing the preceding block once per entity set with a window function rather than per correlated reference; that is sc-120493, with ≤100 ms lock-hold on both backends at batch 500 as the acceptance criterion, using this run's harness and seed. I
 f you'd rather this PR block on that, say so.
   
   <details>
   <summary>Plans for the locked re-check, batch 500, original predicate</summary>
   
   **PostgreSQL 16 — `EXPLAIN (ANALYZE, BUFFERS)`**
   
   ```
   Nested Loop Semi Join  (cost=196.22..51983.18 rows=25 width=16) (actual time=13.530..6626.568 rows=500 loops=1)
     Buffers: shared hit=2631386
     ->  Nested Loop Anti Join  (cost=195.94..13041.32 rows=151 width=75) (actual time=0.586..6.785 rows=500 loops=1)
           Join Filter: ((unresolved_attempt.created_on <= purge_audit_log.created_on) AND ((unresolved_attempt.entity_type)::text = (purge_audit_log.entity_type)::text) AND ((unresolved_attempt.entity_uuid)::text = (purge_audit_log.entity_uuid)::text))
           Buffers: shared hit=2994
           ->  Bitmap Heap Scan on purge_audit_log  (cost=195.65..13029.65 rows=168 width=75) (actual time=0.583..6.041 rows=500 loops=1)
                 Recheck Cond: (id = ANY ('{…500 ids…}'::uuid[]))
                 Filter: ((entity_uuid IS NOT NULL) AND (created_on <= '2026-09-14 16:58:12.052344'::timestamp without time zone) AND ((trigger)::text <> 'force'::text) AND ((status)::text = 'blocked'::text) AND (((SubPlan 1) IS NULL) OR (created_on > (SubPlan 2))))
                 Heap Blocks: exact=11
                 Buffers: shared hit=2992
                 ->  Bitmap Index Scan on purge_audit_log_pkey  (cost=0.00..194.36 rows=500 width=0) (actual time=0.544..0.545 rows=500 loops=1)
                       Index Cond: (id = ANY ('{…500 ids…}'::uuid[]))
                       Buffers: shared hit=123
                 SubPlan 1
                   ->  Aggregate  (cost=12.62..12.63 rows=1 width=8) (actual time=0.004..0.004 rows=1 loops=500)
                         Buffers: shared hit=1465
                         ->  Index Only Scan using ix_purge_audit_log_pruning on purge_audit_log streak_break  (cost=0.29..12.62 rows=1 width=8) (actual time=0.003..0.003 rows=5 loops=500)
                               Index Cond: ((status = ANY ('{target_absent,confirmed}'::text[])) AND (entity_type = (purge_audit_log.entity_type)::text) AND (entity_uuid IS NOT NULL) AND (entity_uuid = (purge_audit_log.entity_uuid)::text) AND (created_on <= '2026-09-14 16:58:12.052344'::timestamp without time zone))
                               Heap Fetches: 464
                               Buffers: shared hit=1465
                 SubPlan 2
                   ->  Aggregate  (cost=12.62..12.63 rows=1 width=8) (actual time=0.004..0.004 rows=1 loops=464)
                         Buffers: shared hit=1393
                         ->  Index Only Scan using ix_purge_audit_log_pruning on purge_audit_log streak_break_1  (cost=0.29..12.62 rows=1 width=8) (actual time=0.002..0.003 rows=5 loops=464)
                               Index Cond: ((status = ANY ('{target_absent,confirmed}'::text[])) AND (entity_type = (purge_audit_log.entity_type)::text) AND (entity_uuid IS NOT NULL) AND (entity_uuid = (purge_audit_log.entity_uuid)::text) AND (created_on <= '2026-09-14 16:58:12.052344'::timestamp without time zone))
                               Heap Fetches: 464
                               Buffers: shared hit=1393
           ->  Materialize  (cost=0.29..8.31 rows=1 width=40) (actual time=0.000..0.000 rows=0 loops=500)
                 Buffers: shared hit=2
                 ->  Index Only Scan using ix_purge_audit_log_pruning on purge_audit_log unresolved_attempt  (cost=0.29..8.30 rows=1 width=40) (actual time=0.002..0.002 rows=0 loops=1)
                       Index Cond: (status = 'pending'::text)
                       Heap Fetches: 0
                       Buffers: shared hit=2
     ->  Index Scan using ix_purge_audit_log_pruning on purge_audit_log earlier_block  (cost=0.29..308.67 rows=1 width=59) (actual time=13.237..13.237 rows=1 loops=500)
           Index Cond: (((status)::text = 'blocked'::text) AND ((entity_type)::text = (purge_audit_log.entity_type)::text) AND ((entity_uuid)::text = (purge_audit_log.entity_uuid)::text) AND (created_on < purge_audit_log.created_on))
           Filter: ((NOT ((reason)::text IS DISTINCT FROM (purge_audit_log.reason)::text)) AND (NOT EXISTS(SubPlan 5)) AND (((SubPlan 3) IS NULL) OR (created_on > (SubPlan 4))))
           Rows Removed by Filter: 4875
           Buffers: shared hit=2628392
           SubPlan 5
             ->  Index Scan using ix_purge_audit_log_pruning on purge_audit_log reason_change  (cost=0.29..8.31 rows=1 width=0) (actual time=0.008..0.008 rows=1 loops=811370)
                   Index Cond: (((status)::text = 'blocked'::text) AND ((entity_type)::text = (purge_audit_log.entity_type)::text) AND ((entity_uuid)::text = (purge_audit_log.entity_uuid)::text) AND (created_on >= earlier_block.created_on) AND (created_on <= purge_audit_log.created_on))
                   Filter: ((reason)::text IS DISTINCT FROM (purge_audit_log.reason)::text)
                   Rows Removed by Filter: 6
                   Buffers: shared hit=2557264
           SubPlan 3
             ->  Aggregate  (cost=12.62..12.63 rows=1 width=8) (actual time=0.012..0.012 rows=1 loops=500)
                   Buffers: shared hit=1465
                   ->  Index Only Scan using ix_purge_audit_log_pruning on purge_audit_log streak_break_2  (cost=0.29..12.62 rows=1 width=8) (actual time=0.008..0.008 rows=5 loops=500)
                         Index Cond: ((status = ANY ('{target_absent,confirmed}'::text[])) AND (entity_type = (earlier_block.entity_type)::text) AND (entity_uuid IS NOT NULL) AND (entity_uuid = (earlier_block.entity_uuid)::text) AND (created_on <= '2026-09-14 16:58:12.052344'::timestamp without time zone))
                         Heap Fetches: 464
                         Buffers: shared hit=1465
           SubPlan 4
             ->  Aggregate  (cost=12.62..12.63 rows=1 width=8) (actual time=0.004..0.004 rows=1 loops=464)
                   Buffers: shared hit=1393
                   ->  Index Only Scan using ix_purge_audit_log_pruning on purge_audit_log streak_break_3  (cost=0.29..12.62 rows=1 width=8) (actual time=0.002..0.003 rows=5 loops=464)
                         Index Cond: ((status = ANY ('{target_absent,confirmed}'::text[])) AND (entity_type = (earlier_block.entity_type)::text) AND (entity_uuid IS NOT NULL) AND (entity_uuid = (earlier_block.entity_uuid)::text) AND (created_on <= '2026-09-14 16:58:12.052344'::timestamp without time zone))
                         Heap Fetches: 464
                         Buffers: shared hit=1393
   Planning:
     Buffers: shared hit=38
   Planning Time: 0.818 ms
   Execution Time: 6626.977 ms
   ```
   
   **MySQL 8.0.44 (REPEATABLE READ) — `EXPLAIN ANALYZE` (tree format)**
   
   ```
   -> Nested loop antijoin  (cost=265 rows=112) (actual time=105..49670 rows=500 loops=1)
       -> Filter: ((purge_audit_log.`status` = 'blocked') and (purge_audit_log.id in (…500 ids…)<?7?','(??;?DI????\'1?@','l????M7?M?e?n^"','e??RS?B??FU????Y','؀D?cI??~ؠ?%Ws','?j?eu?@?Q??Xw8?','?ވKnNJ??gHj?[	','@\Z????H …
           -> Index range scan on purge_audit_log using PRIMARY over (id = 0x0035db15bc4149aabd531a28a3c66c13) OR (id = 0x0053b24e25644ba69bffa365fbe95b63) OR (498 more)  (cost=225 rows=500) (actual time=0.0545..1.23 rows=5 …
           -> Select #2 (subquery in condition; dependent)
               -> Aggregate: max(streak_break.created_on)  (cost=1.45 rows=1) (actual time=0.0105..0.0105 rows=1 loops=500)
                   -> Filter: ((streak_break.entity_type = purge_audit_log.entity_type) and (streak_break.entity_uuid = purge_audit_log.entity_uuid) and (streak_break.entity_uuid is not null) and (streak_break.`status` in ( …
                       -> Covering index range scan on streak_break using ix_purge_audit_log_pruning over (status = 'confirmed') OR (status = 'target_absent')  (cost=1.44 rows=6) (actual time=0.00408..0.00575 rows=5 loops=5 …
           -> Select #3 (subquery in condition; dependent)
               -> Aggregate: max(streak_break.created_on)  (cost=1.45 rows=1) (actual time=0.0103..0.0104 rows=1 loops=464)
                   -> Filter: ((streak_break.entity_type = purge_audit_log.entity_type) and (streak_break.entity_uuid = purge_audit_log.entity_uuid) and (streak_break.entity_uuid is not null) and (streak_break.`status` in ( …
                       -> Covering index range scan on streak_break using ix_purge_audit_log_pruning over (status = 'confirmed') OR (status = 'target_absent')  (cost=1.44 rows=6) (actual time=0.00374..0.00538 rows=5 loops=4 …
           -> Select #4 (subquery in condition; dependent)
               -> Nested loop antijoin  (cost=6.73 rows=1.26) (actual time=22.4..99.3 rows=4.76 loops=500)
                   -> Filter: ((earlier_block.`status` = 'blocked') and (earlier_block.entity_type = purge_audit_log.entity_type) and (((select #5) is null) or (earlier_block.created_on > (select #6))) and (earlier_block.cr …
                       -> Index lookup on earlier_block using ix_purge_audit_log_entity_uuid (entity_uuid=purge_audit_log.entity_uuid)  (cost=6.29 rows=25.2) (actual time=0.0051..4.1 rows=5574 loops=500)
                       -> Select #5 (subquery in condition; dependent)
                           -> Aggregate: max(streak_break.created_on)  (cost=1.45 rows=1) (actual time=0.00607..0.00609 rows=1 loops=2.78e+6)
                               -> Filter: ((streak_break.entity_type = earlier_block.entity_type) and (streak_break.entity_uuid = earlier_block.entity_uuid) and (streak_break.entity_uuid is not null) and (streak_break.`stat …
                                   -> Covering index range scan on streak_break using ix_purge_audit_log_pruning over (status = 'confirmed') OR (status = 'target_absent')  (cost=1.44 rows=6) (actual time=0.00216..0.00365 ro …
                       -> Select #6 (subquery in condition; dependent)
                           -> Aggregate: max(streak_break.created_on)  (cost=1.45 rows=1) (actual time=0.00602..0.00603 rows=1 loops=2.78e+6)
                               -> Filter: ((streak_break.entity_type = earlier_block.entity_type) and (streak_break.entity_uuid = earlier_block.entity_uuid) and (streak_break.entity_uuid is not null) and (streak_break.`stat …
                                   -> Covering index range scan on streak_break using ix_purge_audit_log_pruning over (status = 'confirmed') OR (status = 'target_absent')  (cost=1.44 rows=6) (actual time=0.00212..0.0036 row …
                   -> Filter: ((reason_change.`status` = 'blocked') and (reason_change.created_on >= earlier_block.created_on) and (reason_change.created_on <= purge_audit_log.created_on) and (not((reason_change.reason <=>  …
                       -> Index lookup on reason_change using ix_purge_audit_log_entity_uuid (entity_uuid=purge_audit_log.entity_uuid)  (cost=1424 rows=25.2) (actual time=0.00383..0.222 rows=408 loops=39952)
       -> Filter: ((unresolved_attempt.entity_uuid = purge_audit_log.entity_uuid) and (unresolved_attempt.entity_type = purge_audit_log.entity_type) and (unresolved_attempt.created_on <= purge_audit_log.created_on))  (cost= …
           -> Index lookup on unresolved_attempt using ix_purge_audit_log_status_created_on (status='pending')  (cost=0.251 rows=1) (actual time=0.00342..0.00342 rows=0 loops=500)
   ```
   </details>



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org


Re: [PR] feat(deletion-retention): scheduled pruning policy for the purge audit log [superset]

Posted by "richardfogaca (via GitHub)" <gi...@apache.org>.
richardfogaca merged PR #43490:
URL: https://github.com/apache/superset/pull/43490


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org


Re: [PR] feat(deletion-retention): scheduled pruning policy for the purge audit log [superset]

Posted by "bito-code-review[bot] (via GitHub)" <gi...@apache.org>.
bito-code-review[bot] commented on PR #43490:
URL: https://github.com/apache/superset/pull/43490#issuecomment-5670593048

   AI Code Review is in progress (usually takes 3 to 15 minutes unless it's a very large PR).
   
   <details>
   <summary>
   Bito Usage Guide
   </summary>
   
   **Commands**
         
   Type the following command in the pull request comment and save the comment.
         
   - `/review` - Manually triggers an incremental AI Review.
   - `/review full` - Manually triggers a full AI Review.
   
   - `/pause` - Pauses automatic reviews on this pull request.
   - `/resume` - Resumes automatic reviews.
   - `/resolve` - Marks all Bito-posted review comments as resolved.
   - `/abort` - Cancels all in-progress reviews.
   
         
   Refer to the <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/available-commands" target="_blank" rel="noopener noreferrer">documentation</a> for additional commands.
         
   **Configuration**
   
   This repository uses `Superset` You can customize the agent settings <a href="https://alpha.bito.ai/home/ai-agents/code-review-agent" target="_blank" rel="noopener noreferrer">here</a> or contact your Bito workspace admin at evan@preset.io.
         
   **Documentation & Help**
   - <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/getting-started/install-run-using-bito-cloud/create-or-customize-an-agent-instance" target="_blank" rel="noopener noreferrer">Customize agent settings</a>
   - <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/implementing-custom-code-review-rules" target="_blank" rel="noopener noreferrer">Review rules</a>
   - <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent" target="_blank" rel="noopener noreferrer">General documentation</a>
   - <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/faqs" target="_blank" rel="noopener noreferrer">FAQ</a>
   
   </details>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org


Re: [PR] feat(deletion-retention): scheduled pruning policy for the purge audit log [superset]

Posted by "bito-code-review[bot] (via GitHub)" <gi...@apache.org>.
bito-code-review[bot] commented on code in PR #43490:
URL: https://github.com/apache/superset/pull/43490#discussion_r3855202894


##########
superset/migrations/versions/2026-08-24_15-50_a6c21e5b4d93_index_purge_audit_pruning.py:
##########
@@ -0,0 +1,46 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Index purge-audit pruning predicates.
+
+Revision ID: a6c21e5b4d93
+Revises: 1072de5ed955
+Create Date: 2026-08-24 15:50:00.000000
+
+"""
+
+from superset.migrations.shared.utils import create_index, drop_index
+
+# revision identifiers, used by Alembic.
+revision: str = "a6c21e5b4d93"
+down_revision: str = "1072de5ed955"
+
+_TABLE_NAME: str = "purge_audit_log"
+_INDEX_NAME: str = "ix_purge_audit_log_pruning"
+
+
+def upgrade() -> None:
+    """Add an index matching recurring pruning access patterns."""
+    create_index(
+        _TABLE_NAME,
+        _INDEX_NAME,
+        ["status", "entity_type", "entity_uuid", "created_on"],
+    )
+
+
+def downgrade() -> None:
+    """Remove the purge-audit pruning index."""
+    drop_index(_TABLE_NAME, _INDEX_NAME)

Review Comment:
   <!-- Bito Reply -->
   The user's clarification confirms that the migration is indeed covered by the existing test suite, which validates the table, index name, ordered columns, and downgrade symmetry. The confusion likely stemmed from the filename focusing on the subsequent coordination migration, which may have obscured the index coverage. No further action is required regarding the suggestion for additional unit tests.



##########
tests/unit_tests/commands/deletion_retention/test_prune_audit.py:
##########
@@ -0,0 +1,294 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Unit tests for the pure parts of purge-audit pruning.
+
+Query behavior against real rows (streak survivors, batching, protection
+invariants) is covered by
+``tests/integration_tests/deletion_retention/prune_audit_tests.py``.
+"""
+
+from __future__ import annotations
+
+from contextlib import contextmanager
+from datetime import datetime
+from typing import Any, Iterator
+from unittest.mock import patch
+
+import pytest
+import sqlalchemy as sa
+from flask import current_app
+from sqlalchemy.dialects import mysql
+
+from superset.commands.deletion_retention import prune_audit
+from superset.commands.deletion_retention.prune_audit import (
+    EVIDENCE_RETENTION_KEY,
+    OPERATIONAL_RETENTION_KEY,
+    OPERATIONAL_STATUSES,
+    PROTECTED_STATUSES,
+    PruneRunResult,
+    resolve_evidence_retention_days,
+    resolve_operational_retention_days,
+)
+from superset.models.purge_audit_log import (
+    ALL_STATUSES,
+    STATUS_BLOCKED,
+    STATUS_CONFIRMED,
+    STATUS_FAILED,
+    STATUS_PENDING,
+    STATUS_TARGET_ABSENT,
+)
+
+_METRIC_PREFIX: str = "deletion_retention.prune_purge_audit"
+
+
+@contextmanager
+def without_config(key: str) -> Iterator[None]:
+    """Remove a config key for the duration of the block, then restore it.
+
+    ``patch.dict`` cannot express key *removal*, and the unit-test app
+    fixture is module-scoped, so a bare ``pop`` would leak to later tests.
+    """
+    with patch.dict(current_app.config):
+        current_app.config.pop(key, None)
+        yield
+
+
+def test_retention_categories_partition_every_status() -> None:
+    """Require every status to belong to a retention category.
+
+    A new status added to the model without a category would silently never
+    be pruned; this assertion catches that omission.
+    """
+    assert OPERATIONAL_STATUSES == {STATUS_BLOCKED, STATUS_FAILED}
+    assert PROTECTED_STATUSES == {STATUS_CONFIRMED, STATUS_TARGET_ABSENT}
+    assert not OPERATIONAL_STATUSES & PROTECTED_STATUSES
+    assert OPERATIONAL_STATUSES | PROTECTED_STATUSES | {STATUS_PENDING} == ALL_STATUSES
+
+
+def test_only_proof_of_destruction_breaks_a_blockage_streak() -> None:
+    """Require destruction evidence to break a blockage streak.
+
+    A failed attempt is an infrastructure outcome, and pending is provisional.
+    Neither proves that the blockage cleared.
+    """
+    assert prune_audit._STREAK_BREAKING_STATUSES == {
+        STATUS_CONFIRMED,
+        STATUS_TARGET_ABSENT,
+    }
+    assert STATUS_FAILED not in prune_audit._STREAK_BREAKING_STATUSES
+    assert STATUS_PENDING not in prune_audit._STREAK_BREAKING_STATUSES
+
+
+def test_operational_retention_defaults_to_ninety_days() -> None:
+    """Use the shipped 90-day operational retention default."""
+    assert current_app.config[OPERATIONAL_RETENTION_KEY] == 90
+    assert resolve_operational_retention_days().days == 90
+
+
+@pytest.mark.parametrize("value", [30, 1, 36500])
+def test_operational_retention_accepts_positive_days(value: int) -> None:
+    with patch.dict(current_app.config, {OPERATIONAL_RETENTION_KEY: value}):
+        assert resolve_operational_retention_days() == (value, None)
+
+
+@pytest.mark.parametrize("value", [0, -5, True, False, "ninety", None, 1.5])
+def test_operational_retention_fails_closed_on_invalid_values(value: Any) -> None:
+    """Disable a category and identify its invalid configuration key."""
+    with patch.dict(current_app.config, {OPERATIONAL_RETENTION_KEY: value}):
+        window: prune_audit.ResolvedWindow = resolve_operational_retention_days()
+    assert window.days is None
+    assert window.invalid_key == OPERATIONAL_RETENTION_KEY
+
+
+def test_missing_operational_key_is_reported_as_invalid_not_assumed() -> None:
+    """A popped key is operator error, not a silent 90-day assumption."""
+    with without_config(OPERATIONAL_RETENTION_KEY):
+        window: prune_audit.ResolvedWindow = resolve_operational_retention_days()
+    assert window.days is None
+    assert window.invalid_key == OPERATIONAL_RETENTION_KEY
+
+
+def test_evidence_retention_defaults_to_off_without_warning() -> None:
+    """Unset is the documented never-expire default (FR-006), not an error:
+    disabled, no warning, and no key reported as invalid."""
+    with without_config(EVIDENCE_RETENTION_KEY):
+        with patch.object(prune_audit, "logger") as mock_logger:
+            window: prune_audit.ResolvedWindow = resolve_evidence_retention_days()
+        mock_logger.warning.assert_not_called()
+    assert window == (None, None)
+
+
+def test_evidence_retention_accepts_the_explicit_opt_in() -> None:
+    with patch.dict(current_app.config, {EVIDENCE_RETENTION_KEY: 3650}):
+        window: prune_audit.ResolvedWindow = resolve_evidence_retention_days()
+    assert window.days == 3650
+
+
+@pytest.mark.parametrize("value", [0, -1, True, "forever"])
+def test_evidence_retention_fails_closed_on_invalid_values(value: Any) -> None:
+    with patch.dict(current_app.config, {EVIDENCE_RETENTION_KEY: value}):
+        with patch.object(prune_audit, "logger") as mock_logger:
+            window: prune_audit.ResolvedWindow = resolve_evidence_retention_days()
+        mock_logger.warning.assert_called_once()
+    assert window.days is None
+    assert window.invalid_key == EVIDENCE_RETENTION_KEY
+
+
+def test_prune_run_result_totals_and_dict_shape() -> None:
+    result: PruneRunResult = PruneRunResult(
+        blocked_duplicates=3, operational_expired=2, evidence_expired=1
+    )
+    assert result.total_removed == 6
+    assert result.as_dict() == {
+        "removed": {
+            "blocked_duplicates": 3,
+            "operational_expired": 2,
+            "evidence_expired": 1,
+        },
+        "carried_over": False,
+        "invalid_config_keys": [],
+    }
+
+
+def test_disabled_task_reports_itself_and_removes_nothing() -> None:
+    """Report a disabled run without reaching the prune implementation."""
+    from superset.tasks import deletion_retention as task_module
+
+    with patch.dict(current_app.config, {"PURGE_AUDIT_PRUNING_ENABLED": False}):
+        with (
+            patch.object(task_module, "stats_logger_manager") as mock_stats,
+            patch.object(task_module, "logger") as mock_logger,
+            patch.object(task_module.prune_audit, "run_prune") as mock_run,
+        ):
+            outcome: dict[str, Any] = task_module.prune_purge_audit()
+    assert outcome == {"skipped_disabled": 1}
+    mock_run.assert_not_called()
+    mock_stats.instance.incr.assert_called_once_with(
+        f"{_METRIC_PREFIX}.skipped_disabled"
+    )
+    assert mock_logger.info.called
+
+
+@pytest.mark.parametrize("value", ["false", "0", 1, None])
+def test_non_boolean_master_switch_fails_closed(value: Any) -> None:
+    """Never interpret truthy strings or numeric values as deletion opt-in."""
+    from superset.tasks import deletion_retention as task_module
+
+    with patch.dict(current_app.config, {"PURGE_AUDIT_PRUNING_ENABLED": value}):
+        with (
+            patch.object(task_module, "stats_logger_manager") as mock_stats,
+            patch.object(task_module.prune_audit, "run_prune") as mock_run,
+        ):
+            outcome: dict[str, Any] = task_module.prune_purge_audit()
+
+    assert outcome == {"skipped_invalid_config": 1}
+    mock_run.assert_not_called()
+    mock_stats.instance.incr.assert_called_once_with(
+        f"{_METRIC_PREFIX}.skipped_invalid_config"
+    )
+
+
+def test_failed_run_is_isolated_rolled_back_and_distinguishable() -> None:
+    """Report and isolate a failed pruning run.
+
+    The task rolls back and returns an error marker that cannot be mistaken
+    for a successful run that removed nothing.
+    """
+    from superset.tasks import deletion_retention as task_module
+
+    with patch.dict(current_app.config, {"PURGE_AUDIT_PRUNING_ENABLED": True}):
+        with (
+            patch.object(task_module, "stats_logger_manager") as mock_stats,
+            patch.object(task_module, "db") as mock_db,
+            patch.object(
+                task_module.prune_audit,
+                "run_prune",
+                side_effect=RuntimeError("boom"),
+            ),
+        ):
+            outcome: dict[str, Any] = task_module.prune_purge_audit()
+    assert outcome == {"error": 1}
+    mock_db.session.rollback.assert_called_once()
+    mock_stats.instance.incr.assert_called_once_with(f"{_METRIC_PREFIX}.failed")
+
+
+def test_successful_run_mirrors_counts_and_carryover_into_metrics() -> None:
+    """Expose category counts and convergence through metrics."""
+    from superset.tasks import deletion_retention as task_module
+
+    fake: PruneRunResult = PruneRunResult(
+        blocked_duplicates=7,
+        operational_expired=4,
+        evidence_expired=0,
+        carried_over=True,
+    )

Review Comment:
   <!-- Bito Reply -->
   The user's decision to maintain the local test setups is reasonable. Given that the tests exercise distinct task outcomes and their patch contexts make the boundaries explicit, extracting a shared fixture would indeed introduce unnecessary indirection without providing a significant maintenance benefit. The current approach is acceptable as it keeps the test logic self-contained and clear.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org


Re: [PR] feat(deletion-retention): scheduled pruning policy for the purge audit log [superset]

Posted by "rusackas (via GitHub)" <gi...@apache.org>.
rusackas commented on code in PR #43490:
URL: https://github.com/apache/superset/pull/43490#discussion_r3864759517


##########
superset/initialization/__init__.py:
##########
@@ -1002,11 +1005,14 @@ def _warn_if_retention_beat_missing(self) -> None:
                 "CeleryConfig or add the module to your override.",
                 self._RETENTION_TASK_MODULE,
             )
-        default_flags = self.config.get("DEFAULT_FEATURE_FLAGS", {})
-        configured_flags = self.config.get("FEATURE_FLAGS", {})
-        soft_delete_enabled = bool(
+        default_flags: dict[str, Any] = self.config.get("DEFAULT_FEATURE_FLAGS", {})
+        configured_flags: dict[str, Any] = self.config.get("FEATURE_FLAGS", {})
+        soft_delete_enabled: bool = bool(
             configured_flags.get("SOFT_DELETE", default_flags.get("SOFT_DELETE", False))
         )
+        audit_pruning_enabled: bool = (
+            self.config.get("PURGE_AUDIT_PRUNING_ENABLED", False) is True
+        )

Review Comment:
   @mikebridge did you want to asess/resolve this one, one way or another?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org


Re: [PR] feat(deletion-retention): scheduled pruning policy for the purge audit log [superset]

Posted by "sadpandajoe (via GitHub)" <gi...@apache.org>.
sadpandajoe commented on code in PR #43490:
URL: https://github.com/apache/superset/pull/43490#discussion_r3978079310


##########
UPDATING.md:
##########
@@ -945,7 +980,7 @@ With the flag on, delete confirmations across the chart/dashboard/dataset list p
 
 This also resolves the limitation noted under *Soft delete and restore for datasets*: a database blocked by soft-deleted datasets can now be freed by purging those datasets (per-entity endpoint, retention task, or `force-purge` CLI) instead of hard-deleting `tables` rows out-of-band.
 
-The `purge_audit_log` table is **never pruned by design** — the audit must survive the entities it names; operators who need to age it out should prune manually.
+The `purge_audit_log` table is pruned automatically by the `deletion_retention.prune_purge_audit` Celery beat task (daily, 03:30), so it no longer grows unbounded and does not need manual pruning. The policy is written to preserve the audit's meaning rather than trade it away: within an entity's current blockage streak the earliest — "blocked since" — record always survives (only redundant duplicate `blocked` records are collapsed), and completed-destruction evidence (`confirmed`, `target_absent`) is **never** removed unless the separate `PURGE_AUDIT_EVIDENCE_RETENTION_DAYS` opt-in is explicitly set. What ages out is operational noise — `blocked` records from already-resolved streaks and `failed` records — once older than `PURGE_AUDIT_RETENTION_DAYS` (default 90). Set `PURGE_AUDIT_PRUNING_ENABLED = False` to restore the previous never-pruned, unbounded-growth behavior. See the release-note entry above for the beat-schedule and `CELERY_CONFIG` details.

Review Comment:
   This can leave operators with pruning disabled while believing the table is bounded: the shipped default is `False`, and `PURGE_AUDIT_RETENTION_DAYS` is not read anywhere (the key is `PURGE_AUDIT_OPERATIONAL_RETENTION_DAYS`). Could this state that pruning must be enabled explicitly and use the actual key?



##########
superset/commands/deletion_retention/prune_audit.py:
##########
@@ -0,0 +1,726 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Deletion-only pruning of the ``purge_audit_log`` table.
+
+Bounds the audit history's growth without ever weakening its evidentiary
+value. Three delete categories, applied in priority order under one shared
+per-run batch budget:
+
+1. **Blocked duplicates** — within an entity's *current* blockage streak
+   (its ``blocked`` rows newer than the entity's newest streak-breaking
+   row), only the first row of each run of consecutive same-reason rows
+   survives: the streak's earliest row and the first row after every
+   change of block reason. Later same-reason repeats are removed
+   regardless of age. The survivors carry the "blocked since" fact and
+   the reason history — for coded reasons, the same rows the audit writer's
+   own suppression rule retains (reason-less legacy runs are additionally
+   collapsed to their earliest here) — and are never deleted while the
+   streak is current.
+2. **Operational expiry** — ``blocked`` rows of *resolved* streaks and
+   ``failed`` rows older than ``PURGE_AUDIT_OPERATIONAL_RETENTION_DAYS``
+   age out.
+3. **Evidence expiry** — ``confirmed`` / ``target_absent`` rows are
+   untouchable unless ``PURGE_AUDIT_EVIDENCE_RETENTION_DAYS`` is
+   explicitly set (the operator's compliance assertion), and then only
+   rows older than that window.
+
+``pending`` rows belong to :func:`audit.reconcile_pending` and rows with
+future timestamps (clock skew) are excluded from every category *and* from
+streak classification, so a skewed writer cannot reclassify a live streak.
+
+Three invariants keep the survivor safe without a distributed lock, which
+matters because runs can overlap and both ``reconcile_pending`` and the
+purge path finalize rows concurrently. A duplicate is only ever deleted
+when no concurrent transition could turn it into a survivor first:
+
+* **A boundary is never removed while it still bounds anything.** Evidence
+  expiry refuses to delete a row while an older ``blocked`` or ``pending``
+  row for the same entity survives. Boundaries therefore never *recede*,
+  which would otherwise promote resolved rows into a current streak.
+* **A blocked row whose classification is unstable is never deleted.**
+  Finalizing a ``pending`` row resolves it *in place*, keeping its original
+  timestamp, so an unresolved attempt is a boundary that may appear
+  mid-history at any moment — and the blocked row after it would become its
+  streak's survivor. Blocked rows preceded by an unresolved attempt are
+  therefore skipped by **both** the duplicate and the operational category.
+  Age does not stabilize the classification: an old blocked row inside a
+  live streak is exactly the "blocked for years" case FR-009 protects.
+  Without this, a boundary moving *forward* would demote the current
+  survivor and promote the next row into its place — possibly a row already
+  selected for deletion.
+* **Deletes are conditional and counted from rowcounts.** A row whose
+  status changed since selection is not matched, so overlapping runs can
+  neither double-remove nor double-report.
+
+Boundaries moving forward past *all* of an entity's blocked rows leave no
+survivor to promote, so a selected duplicate may be removed slightly ahead
+of its retention window in that case. It was redundant either way and the
+streak's earliest row is untouched.
+
+Timestamps order rows. Ties are possible — legacy second-precision rows,
+two writers within one clock tick — and are resolved on the preserving
+side: a ``pending`` row tied with a blocked row counts as preceding it (the
+block is deferred until the attempt resolves); a blocked row tied with a
+boundary sits on the boundary's *resolved* side (it ages out instead of
+seeding a new current streak, and the boundary is not removed before it);
+and tied same-reason blocked rows are all retained.
+
+Candidate selection is embedded in each ``DELETE`` statement. The derived-table
+wrapper keeps that shape legal on MySQL while ensuring that a pending or
+recovered row committed before the delete is evaluated participates in the
+survivor and boundary predicates. No stale list of candidate ids crosses a
+transaction boundary.
+
+Audit creation/recovery and every pruning batch take the same singleton
+database write lock before assigning a timestamp or evaluating candidates.
+The lock is held through commit, so an audit row cannot become visible in an
+already-processed logical past. Automatic pruning still ships disabled by
+default so operators explicitly choose their retention policy.
+"""
+
+from __future__ import annotations
+
+import logging
+from collections.abc import Callable
+from dataclasses import dataclass, field
+from datetime import datetime, timedelta
+from functools import partial
+from typing import Any, Literal, NamedTuple, TypeAlias
+
+import sqlalchemy as sa
+from flask import current_app
+
+from superset import db
+from superset.commands.deletion_retention.audit import (
+    acquire_coordination_lock,
+    TRIGGER_FORCE,
+    utc_now,
+)
+from superset.models.purge_audit_log import (
+    PurgeAuditLog,
+    STATUS_BLOCKED,
+    STATUS_CONFIRMED,
+    STATUS_FAILED,
+    STATUS_PENDING,
+    STATUS_TARGET_ABSENT,
+)
+
+logger: logging.Logger = logging.getLogger(__name__)
+
+#: Operational records: noise-prone outcomes whose compliance value decays —
+#: a blocked or failed purge leaves the object in place (FR-001).
+OPERATIONAL_STATUSES: frozenset[str] = frozenset({STATUS_BLOCKED, STATUS_FAILED})
+#: Protected evidence: the only surviving trace of a destroyed object.
+PROTECTED_STATUSES: frozenset[str] = frozenset({STATUS_CONFIRMED, STATUS_TARGET_ABSENT})
+#: The outcomes that end a blockage streak — proof the object is gone.
+#:
+#: ``failed`` is deliberately absent. A failed purge is an infrastructure
+#: outcome (the cascade raised), not evidence the blockage cleared: the
+#: policy that blocked the entity is untouched, so the blockage continues
+#: across it. Treating ``failed`` as a boundary would let one transient
+#: error demote the "blocked since" survivor to an ageing duplicate and
+#: restate the blockage as beginning after the failure — losing exactly the
+#: fact FR-003 exists to preserve. ``pending`` is provisional and likewise
+#: neither joins nor breaks streaks.
+_STREAK_BREAKING_STATUSES: frozenset[str] = frozenset(
+    {STATUS_CONFIRMED, STATUS_TARGET_ABSENT}
+)
+
+#: Rows deleted per statement, matching the purge task's batch convention.
+BATCH_SIZE: int = 500
+#: One shared budget for the whole run across all three categories; the
+#: remaining backlog carries over to the next scheduled run (FR-004/SC-004).
+MAX_BATCHES_PER_RUN: int = 10
+
+OPERATIONAL_RETENTION_KEY: str = "PURGE_AUDIT_OPERATIONAL_RETENTION_DAYS"
+EVIDENCE_RETENTION_KEY: str = "PURGE_AUDIT_EVIDENCE_RETENTION_DAYS"
+
+
+class ResolvedWindow(NamedTuple):
+    """A retention window resolved from config.
+
+    Distinguishes the three outcomes a caller must tell apart: a usable
+    window, the deliberate "off" default, and operator error. Collapsing
+    the last two into a bare ``None`` would force every caller to re-read
+    config to find out which it got.
+    """
+
+    days: int | None
+    invalid_key: str | None = None
+
+
+def _validated_window(key: str, value: Any) -> ResolvedWindow:
+    """Validate a configured day count, failing closed on anything odd.
+
+    An invalid value disables its category for the run rather than widening
+    removal (FR-005/SC-005).
+    """
+    # bool is an int subclass and floats would silently truncate — both are
+    # config mistakes, not day counts, on a knob that deletes rows.
+    if not isinstance(value, bool) and not isinstance(value, float):
+        try:
+            days: int = int(value)
+        except (TypeError, ValueError):
+            days = 0
+        if days > 0:
+            return ResolvedWindow(days)
+    logger.warning(
+        "prune_audit: invalid %s=%r; skipping this category for the run "
+        "(pruning never widens on bad configuration)",
+        key,
+        value,
+    )
+    return ResolvedWindow(None, key)
+
+
+def resolve_operational_retention_days() -> ResolvedWindow:
+    """The operational retention window, or a disabled window when invalid."""
+    return _validated_window(
+        OPERATIONAL_RETENTION_KEY, current_app.config.get(OPERATIONAL_RETENTION_KEY)
+    )
+
+
+def resolve_evidence_retention_days() -> ResolvedWindow:
+    """The evidence expiration window; disabled unless explicitly opted in.
+
+    Unset is the documented "never expire evidence" default (FR-006), not an
+    error, so it produces a disabled window with no warning.
+    """
+    value: Any = current_app.config.get(EVIDENCE_RETENTION_KEY)
+    if value is None:
+        return ResolvedWindow(None)
+    return _validated_window(EVIDENCE_RETENTION_KEY, value)
+
+
+@dataclass
+class PruneRunResult:
+    """Per-category removal counts and run disposition for one pruning run."""
+
+    blocked_duplicates: int = 0
+    operational_expired: int = 0
+    evidence_expired: int = 0
+    #: True when the shared batch budget ran out before every category's
+    #: candidates were drained; the remainder converges on later runs.
+    carried_over: bool = False
+    invalid_config_keys: list[str] = field(default_factory=list)
+
+    @property
+    def total_removed(self) -> int:
+        """Total rows removed across every category."""
+        return (
+            self.blocked_duplicates + self.operational_expired + self.evidence_expired
+        )
+
+    def as_dict(self) -> dict[str, Any]:
+        """The task-return / log-line shape of this result."""
+        return {
+            "removed": {
+                "blocked_duplicates": self.blocked_duplicates,
+                "operational_expired": self.operational_expired,
+                "evidence_expired": self.evidence_expired,
+            },
+            "carried_over": self.carried_over,
+            "invalid_config_keys": list(self.invalid_config_keys),
+        }
+
+
+def _streak_boundary_subquery(now: datetime) -> sa.Subquery:
+    """Per entity, the ``created_on`` of its newest streak-breaking row.
+
+    Entities absent from this subquery have never been proven destroyed —
+    all their blocked rows form one current streak. Future-dated rows are
+    excluded so a skewed writer clock cannot push the boundary ahead of a
+    live streak and make its rows look resolved.
+    """
+    table: sa.Table = PurgeAuditLog.__table__
+    return (
+        sa.select(
+            table.c.entity_type.label("entity_type"),
+            table.c.entity_uuid.label("entity_uuid"),
+            sa.func.max(table.c.created_on).label("boundary"),
+        )
+        .where(table.c.status.in_(_STREAK_BREAKING_STATUSES))
+        .where(table.c.entity_uuid.is_not(None))
+        .where(table.c.created_on <= now)
+        .group_by(table.c.entity_type, table.c.entity_uuid)
+        .subquery("streak_boundary")
+    )
+
+
+def _with_boundary(table: sa.Table, boundary: sa.Subquery) -> sa.Join:
+    """Outer-join each row to its entity's streak boundary (NULL if none)."""
+    return table.outerjoin(
+        boundary,
+        sa.and_(
+            table.c.entity_type == boundary.c.entity_type,
+            table.c.entity_uuid == boundary.c.entity_uuid,
+        ),
+    )
+
+
+def _in_current_streak(
+    row: sa.FromClause, boundary: sa.Subquery
+) -> sa.ColumnElement[bool]:
+    """Whether ``row`` is newer than its entity's boundary (or there is none).
+
+    Strictly newer: a row tied with the boundary sits on its resolved side.
+    A boundary proves the object was gone at that instant, and a block that
+    cannot be ordered after the destruction must not seed a new "current"
+    streak — that would mint a survivor exempt from age-out forever for an
+    object that no longer exists.
+    """
+    return sa.or_(boundary.c.boundary.is_(None), row.c.created_on > boundary.c.boundary)
+
+
+def _repeats_an_earlier_block(
+    table: sa.Table, boundary: sa.Subquery
+) -> sa.ColumnElement[bool]:
+    """Whether a same-reason current-streak block precedes this row with no
+    change of reason in between.
+
+    This is the audit writer's suppression rule
+    (:func:`audit.finalize_retention_blocked`) applied retroactively: a
+    block repeating the reason of the block just before it adds nothing,
+    while the first block after a reason change is the only durable record
+    of the new cause and is retained. The comparison is NULL-safe, so
+    consecutive reason-less (pre-feature) rows count as one run, deduped to
+    the run's earliest, and the first coded block ends that run. Note this
+    is *stricter* than the writer for reason-less rows: the writer never
+    suppresses a reason-less block (``_suppress_redundant_block`` bails on a
+    missing code), so here the pruner additionally collapses legacy
+    pre-feature duplicates — the streak's earliest "blocked since" row is
+    still always kept. Tied same-reason rows are not "earlier" than each
+    other, so all of them are kept.
+
+    A ``force`` row is exempt: it is never reported as a repeat (see the
+    return), so it is kept out of the duplicate category and marked a
+    survivor while its streak is current. It is also exempt from operational
+    age-out (see :func:`_operational_candidates`), so an operator force-purge
+    block is retained permanently — never pruned by either category.
+    """
+    earlier: sa.FromClause = table.alias("earlier_block")
+    between: sa.FromClause = table.alias("reason_change")
+    reason_changed_between: sa.ColumnElement[bool] = sa.exists(
+        sa.select(sa.literal(1))
+        .select_from(between)
+        .where(
+            sa.and_(
+                between.c.status == STATUS_BLOCKED,
+                between.c.entity_type == table.c.entity_type,
+                between.c.entity_uuid == table.c.entity_uuid,
+                # Inclusive bounds: a differing-reason block sharing an
+                # exact timestamp with either endpoint still breaks the run,
+                # so a reason-transition row tied with a neighbour is
+                # preserved as a run head rather than pruned as a repeat
+                # (the same preserving-side tie rule the pending and evidence
+                # guards use). Inclusive bounds only ever add boundaries —
+                # i.e. only ever preserve more, never delete more.
+                between.c.created_on >= earlier.c.created_on,
+                between.c.created_on <= table.c.created_on,
+                between.c.reason.is_distinct_from(table.c.reason),
+            )
+        )
+        .correlate(table, earlier)
+    )
+    repeats: sa.ColumnElement[bool] = sa.exists(
+        sa.select(sa.literal(1))
+        .select_from(earlier)
+        .where(
+            sa.and_(
+                earlier.c.status == STATUS_BLOCKED,
+                earlier.c.entity_type == table.c.entity_type,
+                earlier.c.entity_uuid == table.c.entity_uuid,
+                _in_current_streak(earlier, boundary),
+                earlier.c.created_on < table.c.created_on,
+                earlier.c.reason.is_not_distinct_from(table.c.reason),
+                sa.not_(reason_changed_between),
+            )
+        )
+        .correlate(table, boundary)
+    )
+    # A ``force`` attempt is an operator action the writer never suppresses:
+    # audit.py's ``_suppress_redundant_block`` only collapses consecutive
+    # scheduled same-reason blocks, so the pruner does not collapse a force row
+    # either — it is never reported as a repeat. This keeps it out of the
+    # duplicate category and (via ``sa.not_`` in the operational category) marks
+    # it a survivor while its streak is current. It is also exempt from
+    # operational age-out (``_operational_candidates`` excludes force blocks), so
+    # a force block is retained permanently — full immortality for operator
+    # force-purge blocks. A force row may still be the *earlier* anchor a later
+    # scheduled repeat collapses into — only the force row itself is protected
+    # from duplicate removal.
+    return sa.and_(table.c.trigger != TRIGGER_FORCE, repeats)
+
+
+def _preceded_by_unresolved_attempt(table: sa.Table) -> sa.ColumnElement[bool]:
+    """Whether an unresolved (``pending``) attempt precedes this row.
+
+    A ``pending`` row is the only thing that can insert a streak boundary
+    into *history*: every other write lands at ``now``, newer than every
+    existing row, whereas reconciliation and the purge path finalize a
+    pending row **in place**, keeping its original ``created_on``. So a
+    pending row sitting between two blocked rows is a boundary that may
+    appear at any moment, and the blocked row after it would become the new
+    streak's survivor — the very row pruning must never delete.
+
+    A tied timestamp counts as preceding. Which of the two writes landed
+    first is unknowable from the row, the block's classification (current
+    duplicate, or resolved-streak row on an age window) changes with the
+    attempt's outcome, and deferring it until then costs nothing.
+    """
+    pending: sa.FromClause = table.alias("unresolved_attempt")
+    return sa.exists(
+        sa.select(sa.literal(1))
+        .select_from(pending)
+        .where(
+            sa.and_(
+                pending.c.status == STATUS_PENDING,
+                pending.c.entity_type == table.c.entity_type,
+                pending.c.entity_uuid == table.c.entity_uuid,
+                pending.c.created_on <= table.c.created_on,
+            )
+        )
+    )
+
+
+def _duplicate_candidates(now: datetime, limit: int) -> sa.sql.Select:
+    """Select current-streak blocked rows that repeat the block before them.
+
+    Age-independent by design (FR-003): a repeat is prunable the moment the
+    streak holds an earlier same-reason block, regardless of the retention
+    window. Trigger does not gate streak membership — ``scheduled`` and
+    ``force`` blocked rows share streaks — but a ``force`` row is itself never
+    collapsed as a repeat (see :func:`_repeats_an_earlier_block`). Reason *is*
+    a discriminator: the first block after a reason change survives alongside
+    the streak's earliest row.
+
+    Rows preceded by an unresolved attempt are skipped: their classification
+    is not stable, because that attempt can finalize into a boundary and
+    promote them to survivor between selection and deletion. Such rows are
+    collected once the attempt resolves. Note that pruning does not depend on
+    that happening promptly — reconciliation runs from the purge task, which
+    a deployment may have disabled or left in dry-run — so a long-lived
+    pending row defers its successors indefinitely rather than risking them.
+    """
+    table: sa.Table = PurgeAuditLog.__table__
+    boundary: sa.Subquery = _streak_boundary_subquery(now)
+    return (
+        sa.select(table.c.id)
+        .select_from(_with_boundary(table, boundary))
+        .where(table.c.status == STATUS_BLOCKED)
+        .where(table.c.entity_uuid.is_not(None))
+        .where(table.c.created_on <= now)
+        .where(_in_current_streak(table, boundary))
+        .where(_repeats_an_earlier_block(table, boundary))
+        .where(sa.not_(_preceded_by_unresolved_attempt(table)))
+        .order_by(table.c.created_on)
+        .limit(limit)

Review Comment:
   This limits deletions to 500, but it does not bound the work done while holding the singleton audit lock: once no duplicates remain, the age-unbounded candidate query and its boundary/`EXISTS` checks must examine all matching audit rows before returning zero. On the large tables this feature targets, that can hold the lock long enough for concurrent `write_ahead()` calls to time out and skip purge cycles; could we bound candidate discovery or provide representative query-plan/runtime evidence instead of treating `BATCH_SIZE` as a lock-time bound?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org


Re: [PR] feat(deletion-retention): scheduled pruning policy for the purge audit log [superset]

Posted by "mikebridge (via GitHub)" <gi...@apache.org>.
mikebridge commented on code in PR #43490:
URL: https://github.com/apache/superset/pull/43490#discussion_r3931383394


##########
superset/commands/deletion_retention/prune_audit.py:
##########
@@ -0,0 +1,607 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Deletion-only pruning of the ``purge_audit_log`` table.
+
+Bounds the audit history's growth without ever weakening its evidentiary
+value. Three delete categories, applied in priority order under one shared
+per-run batch budget:
+
+1. **Blocked duplicates** — within an entity's *current* blockage streak
+   (its ``blocked`` rows newer than the entity's newest streak-breaking
+   row), only the streak's earliest row survives: later duplicates are
+   removed regardless of age. The survivor carries the "blocked since"
+   fact and is never deleted while the streak is current.
+2. **Operational expiry** — ``blocked`` rows of *resolved* streaks and
+   ``failed`` rows older than ``PURGE_AUDIT_OPERATIONAL_RETENTION_DAYS``
+   age out.
+3. **Evidence expiry** — ``confirmed`` / ``target_absent`` rows are
+   untouchable unless ``PURGE_AUDIT_EVIDENCE_RETENTION_DAYS`` is
+   explicitly set (the operator's compliance assertion), and then only
+   rows older than that window.
+
+``pending`` rows belong to :func:`audit.reconcile_pending` and rows with
+future timestamps (clock skew) are excluded from every category *and* from
+streak classification, so a skewed writer cannot reclassify a live streak.
+
+Three invariants keep the survivor safe without a distributed lock, which
+matters because runs can overlap and both ``reconcile_pending`` and the
+purge path finalize rows concurrently. A duplicate is only ever deleted
+when no concurrent transition could turn it into a survivor first:
+
+* **A boundary is never removed while it still bounds anything.** Evidence
+  expiry refuses to delete a row while an older ``blocked`` or ``pending``
+  row for the same entity survives. Boundaries therefore never *recede*,
+  which would otherwise promote resolved rows into a current streak.
+* **A blocked row whose classification is unstable is never deleted.**
+  Finalizing a ``pending`` row resolves it *in place*, keeping its original
+  timestamp, so an unresolved attempt is a boundary that may appear
+  mid-history at any moment — and the blocked row after it would become its
+  streak's survivor. Blocked rows preceded by an unresolved attempt are
+  therefore skipped by **both** the duplicate and the operational category.
+  Age does not stabilize the classification: an old blocked row inside a
+  live streak is exactly the "blocked for years" case FR-009 protects.
+  Without this, a boundary moving *forward* would demote the current
+  survivor and promote the next row into its place — possibly a row already
+  selected for deletion.
+* **Deletes are conditional and counted from rowcounts.** A row whose
+  status changed since selection is not matched, so overlapping runs can
+  neither double-remove nor double-report.
+
+Boundaries moving forward past *all* of an entity's blocked rows leave no
+survivor to promote, so a selected duplicate may be removed slightly ahead
+of its retention window in that case. It was redundant either way and the
+streak's earliest row is untouched.
+
+Candidate selection is embedded in each ``DELETE`` statement. The derived-table
+wrapper keeps that shape legal on MySQL while ensuring that a pending or
+recovered row committed before the delete is evaluated participates in the
+survivor and boundary predicates. No stale list of candidate ids crosses a
+transaction boundary.
+
+Audit creation/recovery and every pruning batch take the same singleton
+database write lock before assigning a timestamp or evaluating candidates.
+The lock is held through commit, so an audit row cannot become visible in an
+already-processed logical past. Automatic pruning still ships disabled by
+default so operators explicitly choose their retention policy.
+"""
+
+from __future__ import annotations
+
+import logging
+from collections.abc import Callable
+from dataclasses import dataclass, field
+from datetime import datetime, timedelta
+from functools import partial
+from typing import Any, Literal, NamedTuple, TypeAlias
+
+import sqlalchemy as sa
+from flask import current_app
+
+from superset import db
+from superset.commands.deletion_retention.audit import (
+    acquire_coordination_lock,
+    utc_now,
+)
+from superset.models.purge_audit_log import (
+    PurgeAuditLog,
+    STATUS_BLOCKED,
+    STATUS_CONFIRMED,
+    STATUS_FAILED,
+    STATUS_PENDING,
+    STATUS_TARGET_ABSENT,
+)
+
+logger: logging.Logger = logging.getLogger(__name__)
+
+#: Operational records: noise-prone outcomes whose compliance value decays —
+#: a blocked or failed purge leaves the object in place (FR-001).
+OPERATIONAL_STATUSES: frozenset[str] = frozenset({STATUS_BLOCKED, STATUS_FAILED})
+#: Protected evidence: the only surviving trace of a destroyed object.
+PROTECTED_STATUSES: frozenset[str] = frozenset({STATUS_CONFIRMED, STATUS_TARGET_ABSENT})
+#: The outcomes that end a blockage streak — proof the object is gone.
+#:
+#: ``failed`` is deliberately absent. A failed purge is an infrastructure
+#: outcome (the cascade raised), not evidence the blockage cleared: the
+#: policy that blocked the entity is untouched, so the blockage continues
+#: across it. Treating ``failed`` as a boundary would let one transient
+#: error demote the "blocked since" survivor to an ageing duplicate and
+#: restate the blockage as beginning after the failure — losing exactly the
+#: fact FR-003 exists to preserve. ``pending`` is provisional and likewise
+#: neither joins nor breaks streaks.
+_STREAK_BREAKING_STATUSES: frozenset[str] = frozenset(
+    {STATUS_CONFIRMED, STATUS_TARGET_ABSENT}
+)
+
+#: Rows deleted per statement, matching the purge task's batch convention.
+BATCH_SIZE: int = 500
+#: One shared budget for the whole run across all three categories; the
+#: remaining backlog carries over to the next scheduled run (FR-004/SC-004).
+MAX_BATCHES_PER_RUN: int = 10
+
+OPERATIONAL_RETENTION_KEY: str = "PURGE_AUDIT_OPERATIONAL_RETENTION_DAYS"
+EVIDENCE_RETENTION_KEY: str = "PURGE_AUDIT_EVIDENCE_RETENTION_DAYS"
+
+
+class ResolvedWindow(NamedTuple):
+    """A retention window resolved from config.
+
+    Distinguishes the three outcomes a caller must tell apart: a usable
+    window, the deliberate "off" default, and operator error. Collapsing
+    the last two into a bare ``None`` would force every caller to re-read
+    config to find out which it got.
+    """
+
+    days: int | None
+    invalid_key: str | None = None
+
+
+def _validated_window(key: str, value: Any) -> ResolvedWindow:
+    """Validate a configured day count, failing closed on anything odd.
+
+    An invalid value disables its category for the run rather than widening
+    removal (FR-005/SC-005).
+    """
+    # bool is an int subclass and floats would silently truncate — both are
+    # config mistakes, not day counts, on a knob that deletes rows.
+    if not isinstance(value, bool) and not isinstance(value, float):
+        try:
+            days: int = int(value)
+        except (TypeError, ValueError):
+            days = 0
+        if days > 0:
+            return ResolvedWindow(days)
+    logger.warning(
+        "prune_audit: invalid %s=%r; skipping this category for the run "
+        "(pruning never widens on bad configuration)",
+        key,
+        value,
+    )
+    return ResolvedWindow(None, key)
+
+
+def resolve_operational_retention_days() -> ResolvedWindow:
+    """The operational retention window, or a disabled window when invalid."""
+    return _validated_window(
+        OPERATIONAL_RETENTION_KEY, current_app.config.get(OPERATIONAL_RETENTION_KEY)
+    )
+
+
+def resolve_evidence_retention_days() -> ResolvedWindow:
+    """The evidence expiration window; disabled unless explicitly opted in.
+
+    Unset is the documented "never expire evidence" default (FR-006), not an
+    error, so it produces a disabled window with no warning.
+    """
+    value: Any = current_app.config.get(EVIDENCE_RETENTION_KEY)
+    if value is None:
+        return ResolvedWindow(None)
+    return _validated_window(EVIDENCE_RETENTION_KEY, value)
+
+
+@dataclass
+class PruneRunResult:
+    """Per-category removal counts and run disposition for one pruning run."""
+
+    blocked_duplicates: int = 0
+    operational_expired: int = 0
+    evidence_expired: int = 0
+    #: True when the shared batch budget ran out before every category's
+    #: candidates were drained; the remainder converges on later runs.
+    carried_over: bool = False
+    invalid_config_keys: list[str] = field(default_factory=list)
+
+    @property
+    def total_removed(self) -> int:
+        """Total rows removed across every category."""
+        return (
+            self.blocked_duplicates + self.operational_expired + self.evidence_expired
+        )
+
+    def as_dict(self) -> dict[str, Any]:
+        """The task-return / log-line shape of this result."""
+        return {
+            "removed": {
+                "blocked_duplicates": self.blocked_duplicates,
+                "operational_expired": self.operational_expired,
+                "evidence_expired": self.evidence_expired,
+            },
+            "carried_over": self.carried_over,
+            "invalid_config_keys": list(self.invalid_config_keys),
+        }
+
+
+def _streak_boundary_subquery(now: datetime) -> sa.Subquery:
+    """Per entity, the ``created_on`` of its newest streak-breaking row.
+
+    Entities absent from this subquery have never been proven destroyed —
+    all their blocked rows form one current streak. Future-dated rows are
+    excluded so a skewed writer clock cannot push the boundary ahead of a
+    live streak and make its rows look resolved.
+    """
+    table: sa.Table = PurgeAuditLog.__table__
+    return (
+        sa.select(
+            table.c.entity_type.label("entity_type"),
+            table.c.entity_uuid.label("entity_uuid"),
+            sa.func.max(table.c.created_on).label("boundary"),
+        )
+        .where(table.c.status.in_(_STREAK_BREAKING_STATUSES))
+        .where(table.c.entity_uuid.is_not(None))
+        .where(table.c.created_on <= now)
+        .group_by(table.c.entity_type, table.c.entity_uuid)
+        .subquery("streak_boundary")
+    )
+
+
+def _survivor_subquery(now: datetime) -> sa.Subquery:
+    """Per entity, the ``created_on`` of its current streak's earliest row.
+
+    The survivor is the earliest ``blocked`` row strictly newer than the
+    entity's boundary (or its earliest blocked row outright when no boundary
+    exists). Rows sharing that exact timestamp are all treated as survivors —
+    with microsecond precision ties are pathological, and keeping an extra
+    row errs on the preserving side.
+    """
+    table: sa.Table = PurgeAuditLog.__table__
+    boundary: sa.Subquery = _streak_boundary_subquery(now)
+    return (
+        sa.select(
+            table.c.entity_type.label("entity_type"),
+            table.c.entity_uuid.label("entity_uuid"),
+            sa.func.min(table.c.created_on).label("survivor_created_on"),
+        )
+        .select_from(
+            table.outerjoin(
+                boundary,
+                sa.and_(
+                    table.c.entity_type == boundary.c.entity_type,
+                    table.c.entity_uuid == boundary.c.entity_uuid,
+                ),
+            )
+        )
+        .where(table.c.status == STATUS_BLOCKED)
+        .where(table.c.entity_uuid.is_not(None))
+        .where(table.c.created_on <= now)
+        .where(
+            sa.or_(
+                boundary.c.boundary.is_(None),
+                table.c.created_on > boundary.c.boundary,
+            )
+        )
+        .group_by(table.c.entity_type, table.c.entity_uuid)
+        .subquery("streak_survivor")
+    )
+
+
+def _preceded_by_unresolved_attempt(table: sa.Table) -> sa.ColumnElement[bool]:
+    """Whether an unresolved (``pending``) attempt precedes this row.
+
+    A ``pending`` row is the only thing that can insert a streak boundary
+    into *history*: every other write lands at ``now``, newer than every
+    existing row, whereas reconciliation and the purge path finalize a
+    pending row **in place**, keeping its original ``created_on``. So a
+    pending row sitting between two blocked rows is a boundary that may
+    appear at any moment, and the blocked row after it would become the new
+    streak's survivor — the very row pruning must never delete.
+    """
+    pending: sa.Table = table.alias("unresolved_attempt")
+    return sa.exists(
+        sa.select(sa.literal(1))
+        .select_from(pending)
+        .where(
+            sa.and_(
+                pending.c.status == STATUS_PENDING,
+                pending.c.entity_type == table.c.entity_type,
+                pending.c.entity_uuid == table.c.entity_uuid,
+                pending.c.created_on < table.c.created_on,

Review Comment:
   Verified COVERED — thanks for the MySQL second-granularity angle, that's exactly the tie worth pinning. Built your scenario as an integration test against current head:
   
   `blocked(T0) → pending(T1) → blocked(T1)` with the pending sharing the later block's **exact** `created_on` (seeded via `at=`, simulating MySQL second granularity). The later block **survives**: `_preceded_by_unresolved_attempt` uses `pending.created_on <= table.created_on`, so a tied pending counts as preceding, and the block is skipped by both the duplicate and operational categories.
   
   Evidence (`test_pending_tied_to_later_block_protects_it`, in `ac9bb3bf34`):
   - Against head: passes (b1 retained).
   - Reverted-fix control: change that `<=` to `<` and the test fails (b1 pruned) — confirming the inclusive tie is load-bearing, not incidental.
   
   So the same-timestamp case is handled; the guard is the `<=`. Test added so it stays that way.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org


Re: [PR] feat(deletion-retention): scheduled pruning policy for the purge audit log [superset]

Posted by "mikebridge (via GitHub)" <gi...@apache.org>.
mikebridge commented on PR #43490:
URL: https://github.com/apache/superset/pull/43490#issuecomment-5536525332

   ⚠️ **Please do not merge on the current green badge** — a fix landed after it.
   
   @sadpandajoe raised three over-deletion findings on the deletion predicate. Adjudicated each by building the exact scenario as an integration test against head, with reverted-fix controls (replies on each thread):
   
   - **Finding 1** (pending tied to the later block at MySQL second granularity): **COVERED** — the `<=` unresolved-attempt tie protects it; control proves it load-bearing.
   - **Finding 2** (reason transition amid repeats): **COVERED** — the reason discrimination preserves the transition; control proves it.
   - **Finding 3** (force-purge record erased as a duplicate): **REAL over-deletion bug** — now fixed in `ac9bb3bf34`. `_repeats_an_earlier_block` excludes force-trigger rows so a force attempt is never collapsed. Added tests; 34 unit + 117 integration green; changed-file pre-commit clean.
   
   Because `ac9bb3bf34` adds a real deletion-logic fix on top of the previously-approved head, the approval is stale — the new commit needs a fresh committer look. There is also **one open policy call for the committer** on thread 3: a force block in a *resolved* streak still ages on the operational window (taxonomy-consistent; scoped in the docs and pinned by a test). If it should instead be fully immortal, that's a one-line extension I'll make on your word.
   
   A full 4-lens review of the fix (python/sqlalchemy/clean-code/tidy-first) is filed in the spec repo.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org


Re: [PR] feat(deletion-retention): scheduled pruning policy for the purge audit log [superset]

Posted by "bito-code-review[bot] (via GitHub)" <gi...@apache.org>.
bito-code-review[bot] commented on PR #43490:
URL: https://github.com/apache/superset/pull/43490#issuecomment-5538412197

   AI Code Review is in progress (usually takes 3 to 15 minutes unless it's a very large PR).
   
   <details>
   <summary>
   Bito Usage Guide
   </summary>
   
   **Commands**
         
   Type the following command in the pull request comment and save the comment.
         
   - `/review` - Manually triggers an incremental AI Review.
   - `/review full` - Manually triggers a full AI Review.
   
   - `/pause` - Pauses automatic reviews on this pull request.
   - `/resume` - Resumes automatic reviews.
   - `/resolve` - Marks all Bito-posted review comments as resolved.
   - `/abort` - Cancels all in-progress reviews.
   
         
   Refer to the <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/available-commands" target="_blank" rel="noopener noreferrer">documentation</a> for additional commands.
         
   **Configuration**
   
   This repository uses `Superset` You can customize the agent settings <a href="https://alpha.bito.ai/home/ai-agents/code-review-agent" target="_blank" rel="noopener noreferrer">here</a> or contact your Bito workspace admin at evan@preset.io.
         
   **Documentation & Help**
   - <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/getting-started/install-run-using-bito-cloud/create-or-customize-an-agent-instance" target="_blank" rel="noopener noreferrer">Customize agent settings</a>
   - <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/implementing-custom-code-review-rules" target="_blank" rel="noopener noreferrer">Review rules</a>
   - <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent" target="_blank" rel="noopener noreferrer">General documentation</a>
   - <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/faqs" target="_blank" rel="noopener noreferrer">FAQ</a>
   
   </details>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org


Re: [PR] feat(deletion-retention): scheduled pruning policy for the purge audit log [superset]

Posted by "sadpandajoe (via GitHub)" <gi...@apache.org>.
sadpandajoe commented on code in PR #43490:
URL: https://github.com/apache/superset/pull/43490#discussion_r3940401231


##########
UPDATING.md:
##########
@@ -63,6 +63,41 @@ payload. Clients must display the new impact and obtain renewed confirmation
 before retrying. Preview or recheck failures fail closed rather than treating
 unknown impact as zero. Chart and dashboard purge endpoints are unchanged.
 
+- The purge audit log can now be pruned automatically. The new

Review Comment:
   This adds automatic pruning while the same operator guide still says `purge_audit_log` is never pruned and must be pruned manually at line 803. That leaves two incompatible retention policies for operators; should the older statement be revised or removed?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org


Re: [PR] feat(deletion-retention): scheduled pruning policy for the purge audit log [superset]

Posted by "mikebridge (via GitHub)" <gi...@apache.org>.
mikebridge commented on code in PR #43490:
URL: https://github.com/apache/superset/pull/43490#discussion_r3855201209


##########
superset/migrations/versions/2026-08-24_15-50_a6c21e5b4d93_index_purge_audit_pruning.py:
##########
@@ -0,0 +1,46 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Index purge-audit pruning predicates.
+
+Revision ID: a6c21e5b4d93
+Revises: 1072de5ed955
+Create Date: 2026-08-24 15:50:00.000000
+
+"""
+
+from superset.migrations.shared.utils import create_index, drop_index
+
+# revision identifiers, used by Alembic.
+revision: str = "a6c21e5b4d93"
+down_revision: str = "1072de5ed955"
+
+_TABLE_NAME: str = "purge_audit_log"
+_INDEX_NAME: str = "ix_purge_audit_log_pruning"
+
+
+def upgrade() -> None:
+    """Add an index matching recurring pruning access patterns."""
+    create_index(
+        _TABLE_NAME,
+        _INDEX_NAME,
+        ["status", "entity_type", "entity_uuid", "created_on"],
+    )
+
+
+def downgrade() -> None:
+    """Remove the purge-audit pruning index."""
+    drop_index(_TABLE_NAME, _INDEX_NAME)

Review Comment:
   Thanks — this migration is covered in `tests/unit_tests/migrations/test_purge_audit_coordination.py`. `test_index_upgrade_creates_pruning_index` verifies the table, index name, and ordered columns, and `test_index_downgrade_drops_pruning_index` verifies downgrade symmetry. The file also covers the immediately-following coordination migration; its filename emphasizes that migration, which may have made the index coverage easy to miss.



##########
tests/integration_tests/deletion_retention/prune_audit_tests.py:
##########
@@ -0,0 +1,484 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Integration tests for purge-audit pruning against the real metadata DB.
+
+Rows are seeded directly (pruning reads only ``purge_audit_log``); every
+test's rows carry a distinctive ``entity_type`` prefix and are removed in
+teardown, so no other suite ever sees them. Scoping on ``entity_type``
+rather than ``entity_uuid`` matters: some rows deliberately carry a NULL
+uuid, which a ``LIKE`` on the uuid column would never match.
+"""
+
+from __future__ import annotations
+
+from collections.abc import Callable
+from datetime import timedelta
+from functools import partial
+from typing import Any
+from unittest.mock import patch
+from uuid import UUID, uuid4
+
+import sqlalchemy as sa
+from flask import current_app
+from sqlalchemy.orm import Query
+
+from superset import db
+from superset.commands.deletion_retention import audit, prune_audit
+from superset.commands.deletion_retention.prune_audit import (
+    EVIDENCE_RETENTION_KEY,
+    OPERATIONAL_RETENTION_KEY,
+)
+from superset.models.purge_audit_log import (
+    PurgeAuditLog,
+    STATUS_BLOCKED,
+    STATUS_CONFIRMED,
+    STATUS_FAILED,
+    STATUS_PENDING,
+    STATUS_TARGET_ABSENT,
+)
+from tests.integration_tests.base_tests import SupersetTestCase
+from tests.integration_tests.deletion_retention._base import (
+    ensure_purge_audit_coordination,
+)
+
+_PREFIX: str = "prune_audit_it_"
+_ENTITY_TYPE: str = f"{_PREFIX}slices"
+
+
+class TestPruneAudit(SupersetTestCase):
+    """Behavioral coverage for ``prune_audit.run_prune``."""
+
+    _seq: int
+
+    def setUp(self) -> None:
+        super().setUp()
+        ensure_purge_audit_coordination()
+        self._seq = 0
+        self._cleanup()
+
+    def tearDown(self) -> None:
+        self._cleanup()
+        super().tearDown()
+
+    def _cleanup(self) -> None:
+        db.session.rollback()
+        db.session.execute(
+            sa.delete(PurgeAuditLog.__table__).where(
+                PurgeAuditLog.__table__.c.entity_type == _ENTITY_TYPE
+            )
+        )
+        db.session.commit()
+
+    def add_row(
+        self,
+        status: str,
+        entity: str | None = "e1",
+        age_days: float = 0,
+        trigger: str = audit.TRIGGER_RETENTION,
+    ) -> UUID:
+        """Seed one audit row and return its id.
+
+        Ids rather than instances: an instance whose row a later prune
+        deletes raises ObjectDeletedError on attribute access. ``age_days``
+        may be fractional; a strictly increasing microsecond sequence keeps
+        every ``created_on`` unique so streak ordering is deterministic.
+        ``entity=None`` seeds a row with no ``entity_uuid``.
+        """
+        self._seq += 1
+        row: PurgeAuditLog = PurgeAuditLog(
+            id=uuid4(),
+            status=status,
+            trigger=trigger,
+            actor=audit.ACTOR_SYSTEM,
+            entity_type=_ENTITY_TYPE,
+            entity_uuid=None if entity is None else f"{_PREFIX}{entity}",
+            created_on=audit.utc_now()
+            - timedelta(days=age_days)
+            + timedelta(microseconds=self._seq),
+        )
+        db.session.add(row)
+        db.session.commit()
+        return row.id
+
+    def remaining_ids(self, entity: str | None = "__any__") -> list[UUID]:
+        """Ids of surviving seeded rows, oldest first."""
+        query: Query[PurgeAuditLog] = db.session.query(PurgeAuditLog).filter(
+            PurgeAuditLog.entity_type == _ENTITY_TYPE
+        )
+        if entity != "__any__":
+            query = query.filter(
+                PurgeAuditLog.entity_uuid.is_(None)
+                if entity is None
+                else PurgeAuditLog.entity_uuid == f"{_PREFIX}{entity}"
+            )
+        return [r.id for r in query.order_by(PurgeAuditLog.created_on).all()]
+
+    def run_prune(self, **config: Any) -> prune_audit.PruneRunResult:
+        with patch.dict(current_app.config, config):
+            return prune_audit.run_prune()
+
+    # -- US1: bounded blockage history -------------------------------------
+
+    def test_duplicates_reduce_to_the_earliest_survivor_regardless_of_age(self) -> None:
+        """Keep only the earliest row in a current blockage streak."""
+        ids: list[UUID] = [
+            self.add_row(STATUS_BLOCKED, age_days=5 - i / 100) for i in range(50)
+        ]
+
+        result: prune_audit.PruneRunResult = self.run_prune()
+
+        assert result.blocked_duplicates == 49
+        assert result.carried_over is False
+        assert self.remaining_ids("e1") == [ids[0]]
+
+    def test_backlog_converges_over_bounded_runs(self) -> None:
+        """Drain oversized backlogs across bounded successive runs."""
+        for i in range(56):
+            self.add_row(STATUS_BLOCKED, age_days=3 - i / 1000)
+
+        removed_per_run: list[int] = []
+        with (
+            patch.object(prune_audit, "BATCH_SIZE", 10),
+            patch.object(prune_audit, "MAX_BATCHES_PER_RUN", 2),
+        ):
+            while True:
+                result: prune_audit.PruneRunResult = self.run_prune()
+                removed_per_run.append(result.blocked_duplicates)
+                if not result.carried_over:
+                    break
+
+        assert sum(removed_per_run) == 55  # every duplicate, none of the survivor
+        assert all(n <= 20 for n in removed_per_run)  # never exceeds the budget
+        assert removed_per_run[-1] > 0  # carryover is not reported spuriously
+        assert len(self.remaining_ids("e1")) == 1  # the survivor
+
+    def test_dedup_never_crosses_entities(self) -> None:
+        a_ids: list[UUID] = [
+            self.add_row(STATUS_BLOCKED, entity="a", age_days=2) for _ in range(3)
+        ]
+        b_ids: list[UUID] = [
+            self.add_row(STATUS_BLOCKED, entity="b", age_days=2) for _ in range(4)
+        ]
+
+        result: prune_audit.PruneRunResult = self.run_prune()
+
+        assert result.blocked_duplicates == 5
+        assert self.remaining_ids("a") == [a_ids[0]]
+        assert self.remaining_ids("b") == [b_ids[0]]
+
+    def test_resolved_streak_has_no_survivor_and_ages_out(self) -> None:
+        """Expire blocked rows after newer evidence resolves their streak."""
+        for _ in range(3):
+            self.add_row(STATUS_BLOCKED, age_days=200)
+        confirmed: UUID = self.add_row(STATUS_CONFIRMED, age_days=100)
+        new_survivor: UUID = self.add_row(STATUS_BLOCKED, age_days=50)
+        self.add_row(STATUS_BLOCKED, age_days=40)  # duplicate in the new streak
+
+        result: prune_audit.PruneRunResult = self.run_prune()
+
+        assert result.blocked_duplicates == 1
+        assert result.operational_expired == 3
+        assert set(self.remaining_ids("e1")) == {confirmed, new_survivor}
+
+    def test_a_failed_attempt_does_not_reset_the_blocked_since_survivor(self) -> None:
+        """Keep one blockage streak across a failed purge attempt."""
+        blocked_since: UUID = self.add_row(STATUS_BLOCKED, age_days=200)
+        self.add_row(STATUS_FAILED, age_days=150)  # ages out as operational
+        self.add_row(STATUS_BLOCKED, age_days=100)
+        self.add_row(STATUS_BLOCKED, age_days=50)
+
+        result: prune_audit.PruneRunResult = self.run_prune()
+
+        assert result.blocked_duplicates == 2
+        assert result.operational_expired == 1
+        assert self.remaining_ids("e1") == [blocked_since]
+
+    # -- US2: evidence survives by default ----------------------------------
+
+    def test_defaults_leave_completed_destruction_evidence_untouched(self) -> None:
+        evidence: list[UUID] = [
+            self.add_row(STATUS_CONFIRMED, entity="ev", age_days=3650),
+            self.add_row(STATUS_TARGET_ABSENT, entity="ev", age_days=1000),
+            self.add_row(STATUS_CONFIRMED, entity="ev", age_days=1),
+        ]
+        self.add_row(STATUS_FAILED, entity="ev", age_days=365)  # ages out
+
+        for _ in range(3):  # any sequence of runs (SC-002)
+            result: prune_audit.PruneRunResult = self.run_prune()
+
+        assert result.evidence_expired == 0
+        assert set(self.remaining_ids("ev")) == set(evidence)
+
+    def test_evidence_opt_in_expires_only_rows_older_than_its_window(self) -> None:
+        old: UUID = self.add_row(STATUS_CONFIRMED, entity="ev", age_days=400)
+        young: UUID = self.add_row(STATUS_TARGET_ABSENT, entity="ev", age_days=300)
+
+        result: prune_audit.PruneRunResult = self.run_prune(
+            **{EVIDENCE_RETENTION_KEY: 365}
+        )
+
+        assert result.evidence_expired == 1
+        assert self.remaining_ids("ev") == [young]
+        assert old not in self.remaining_ids("ev")
+
+    def test_opt_in_disabled_again_removes_no_further_evidence(self) -> None:
+        self.add_row(STATUS_CONFIRMED, entity="ev", age_days=400)
+        survivor: UUID = self.add_row(STATUS_CONFIRMED, entity="ev", age_days=390)
+
+        first: prune_audit.PruneRunResult = self.run_prune(
+            **{EVIDENCE_RETENTION_KEY: 395}
+        )
+        assert first.evidence_expired == 1
+
+        second: prune_audit.PruneRunResult = self.run_prune(
+            **{EVIDENCE_RETENTION_KEY: None}
+        )
+        assert second.evidence_expired == 0
+        assert self.remaining_ids("ev") == [survivor]
+
+    def test_pending_and_future_rows_survive_every_configuration(self) -> None:
+        pending: UUID = self.add_row(STATUS_PENDING, entity="px", age_days=3650)
+        future_blocked: UUID = self.add_row(STATUS_BLOCKED, entity="px", age_days=-1)
+        future_confirmed: UUID = self.add_row(
+            STATUS_CONFIRMED, entity="px", age_days=-2
+        )
+
+        result: prune_audit.PruneRunResult = self.run_prune(
+            **{OPERATIONAL_RETENTION_KEY: 1, EVIDENCE_RETENTION_KEY: 1}
+        )
+
+        assert result.total_removed == 0
+        assert set(self.remaining_ids("px")) == {
+            pending,
+            future_blocked,
+            future_confirmed,
+        }
+
+    def test_a_future_dated_outcome_cannot_resolve_a_live_streak(self) -> None:
+        """Exclude future-dated evidence from current streak boundaries."""
+        blocked_since: UUID = self.add_row(STATUS_BLOCKED, entity="sk", age_days=500)
+        self.add_row(STATUS_BLOCKED, entity="sk", age_days=400)
+        skewed: UUID = self.add_row(STATUS_CONFIRMED, entity="sk", age_days=-30)
+
+        result: prune_audit.PruneRunResult = self.run_prune()
+
+        assert result.blocked_duplicates == 1
+        assert result.operational_expired == 0
+        assert set(self.remaining_ids("sk")) == {blocked_since, skewed}
+
+    # -- Survivor invariant under boundary removal (the review's HIGH) ------
+
+    def test_evidence_expiry_spares_a_boundary_that_still_bounds_blocked_rows(
+        self,
+    ) -> None:
+        """Keep evidence while it still bounds surviving blocked rows."""
+        blocked: list[UUID] = [
+            self.add_row(STATUS_BLOCKED, entity="bd", age_days=30) for _ in range(3)
+        ]
+        boundary: UUID = self.add_row(STATUS_CONFIRMED, entity="bd", age_days=20)
+
+        # Evidence window far shorter than the operational one: the boundary
+        # is past its cutoff, the blocked rows it resolved are not.
+        first: prune_audit.PruneRunResult = self.run_prune(
+            **{OPERATIONAL_RETENTION_KEY: 90, EVIDENCE_RETENTION_KEY: 1}
+        )
+
+        assert first.evidence_expired == 0
+        assert set(self.remaining_ids("bd")) == {*blocked, boundary}
+
+        # Once the blocked rows age out, the boundary bounds nothing and
+        # becomes expirable — the guard defers, it does not immortalize.
+        # Categories drain in order within one run, so the blocked rows go
+        # first and the boundary follows in the same pass.
+        second: prune_audit.PruneRunResult = self.run_prune(
+            **{OPERATIONAL_RETENTION_KEY: 10, EVIDENCE_RETENTION_KEY: 1}
+        )
+        assert second.operational_expired == 3
+        assert second.evidence_expired == 1
+        assert self.remaining_ids("bd") == []
+
+    def test_a_block_after_an_unresolved_attempt_is_not_treated_as_a_duplicate(
+        self,
+    ) -> None:
+        """An in-flight attempt is a boundary waiting to happen.
+
+        Finalizing a ``pending`` row resolves it *in place*, keeping its
+        original timestamp — the one way a boundary can appear in the middle
+        of history. The blocked row after it would become the new streak's
+        survivor, so it must not be classified as a duplicate while the
+        attempt is unresolved. Deleting it would destroy the only record
+        that the entity was still blocked after that attempt.
+        """
+        blocked_since: UUID = self.add_row(STATUS_BLOCKED, entity="ua", age_days=100)
+        attempt: UUID = self.add_row(STATUS_PENDING, entity="ua", age_days=50)
+        later_block: UUID = self.add_row(STATUS_BLOCKED, entity="ua", age_days=10)
+
+        result: prune_audit.PruneRunResult = self.run_prune()
+
+        assert result.blocked_duplicates == 0
+        assert set(self.remaining_ids("ua")) == {blocked_since, attempt, later_block}
+
+        # Once the attempt finalizes, the boundary is real: the later block
+        # is the new streak's survivor and stays; the older block is now a
+        # resolved-streak row that ages out on the normal window.
+        db.session.execute(
+            sa.update(PurgeAuditLog.__table__)
+            .where(PurgeAuditLog.__table__.c.id == attempt)
+            .values(status=STATUS_CONFIRMED)
+        )
+        db.session.commit()
+
+        after: prune_audit.PruneRunResult = self.run_prune()
+
+        assert after.blocked_duplicates == 0
+        assert after.operational_expired == 1  # the pre-attempt block
+        assert set(self.remaining_ids("ua")) == {attempt, later_block}
+
+    def test_age_does_not_make_an_unstable_block_expirable(self) -> None:
+        """Keep aged blocked rows whose classification remains unstable.
+
+        Seeded entirely outside the retention window, so only the guard —
+        not the cutoff — can save the row.
+        """
+        blocked_since: UUID = self.add_row(STATUS_BLOCKED, entity="ao", age_days=100)
+        attempt: UUID = self.add_row(STATUS_PENDING, entity="ao", age_days=98)
+        later_block: UUID = self.add_row(STATUS_BLOCKED, entity="ao", age_days=95)
+
+        result: prune_audit.PruneRunResult = self.run_prune()
+
+        assert result.blocked_duplicates == 0
+        assert result.operational_expired == 0
+        assert set(self.remaining_ids("ao")) == {blocked_since, attempt, later_block}
+
+        # Resolving the attempt makes the boundary real: the later block is
+        # the current streak's survivor and is exempt regardless of age,
+        # while the pre-attempt block ages out.
+        db.session.execute(
+            sa.update(PurgeAuditLog.__table__)
+            .where(PurgeAuditLog.__table__.c.id == attempt)
+            .values(status=STATUS_TARGET_ABSENT)
+        )
+        db.session.commit()
+
+        after: prune_audit.PruneRunResult = self.run_prune()
+
+        assert after.operational_expired == 1
+        assert set(self.remaining_ids("ao")) == {attempt, later_block}
+
+    def test_evidence_guard_also_defers_to_an_unresolved_older_attempt(self) -> None:
+        """Treat an older pending attempt as potential blocked evidence."""
+        attempt: UUID = self.add_row(STATUS_PENDING, entity="ug", age_days=400)
+        boundary: UUID = self.add_row(STATUS_CONFIRMED, entity="ug", age_days=300)
+
+        result: prune_audit.PruneRunResult = self.run_prune(
+            **{EVIDENCE_RETENTION_KEY: 200}
+        )
+
+        assert result.evidence_expired == 0
+        assert set(self.remaining_ids("ug")) == {attempt, boundary}
+
+    def test_blocked_rows_without_an_entity_uuid_are_never_aged_out(self) -> None:
+        """Keep UUID-less blocked rows that cannot be proven redundant."""
+        anonymous_block: UUID = self.add_row(STATUS_BLOCKED, entity=None, age_days=1000)
+        self.add_row(STATUS_FAILED, entity=None, age_days=1000)
+
+        result: prune_audit.PruneRunResult = self.run_prune()
+
+        assert result.operational_expired == 1
+        assert self.remaining_ids(None) == [anonymous_block]
+
+    # -- US3: operator controls and observability ---------------------------
+
+    def test_configured_retention_window_is_honored(self) -> None:
+        kept: UUID = self.add_row(STATUS_FAILED, entity="w", age_days=5)
+        self.add_row(STATUS_FAILED, entity="w", age_days=15)
+
+        result: prune_audit.PruneRunResult = self.run_prune(
+            **{OPERATIONAL_RETENTION_KEY: 10}
+        )
+
+        assert result.operational_expired == 1
+        assert self.remaining_ids("w") == [kept]
+
+    def test_invalid_operational_window_skips_the_category_not_widens(self) -> None:
+        """Fail closed when the operational retention window is invalid."""
+        old_failed: UUID = self.add_row(STATUS_FAILED, entity="iv", age_days=1000)
+        self.add_row(STATUS_BLOCKED, entity="iv", age_days=3)
+        self.add_row(STATUS_BLOCKED, entity="iv", age_days=2)
+
+        result: prune_audit.PruneRunResult = self.run_prune(
+            **{OPERATIONAL_RETENTION_KEY: 0}
+        )
+
+        assert result.operational_expired == 0
+        assert result.blocked_duplicates == 1
+        assert result.invalid_config_keys == [OPERATIONAL_RETENTION_KEY]
+        assert old_failed in self.remaining_ids("iv")
+
+    def test_second_run_over_the_same_candidates_removes_and_reports_zero(self) -> None:
+        """Report zero when rerunning over an already-pruned history."""
+        for _ in range(4):
+            self.add_row(STATUS_BLOCKED, entity="cc", age_days=2)
+        self.add_row(STATUS_FAILED, entity="cc", age_days=100)
+
+        first: prune_audit.PruneRunResult = self.run_prune()
+        assert first.blocked_duplicates == 3
+        assert first.operational_expired == 1
+
+        second: prune_audit.PruneRunResult = self.run_prune()
+        assert second.total_removed == 0
+        assert len(self.remaining_ids("cc")) == 1
+
+    def test_a_duplicate_backlog_cannot_starve_the_age_based_categories(self) -> None:
+        """Prevent duplicate backlogs from starving age-based categories."""
+        for _ in range(20):
+            self.add_row(STATUS_BLOCKED, entity="st", age_days=2)
+        old_failed: UUID = self.add_row(STATUS_FAILED, entity="st", age_days=1000)
+
+        with (
+            patch.object(prune_audit, "BATCH_SIZE", 1),
+            patch.object(prune_audit, "MAX_BATCHES_PER_RUN", 2),
+        ):
+            result: prune_audit.PruneRunResult = self.run_prune()
+
+        assert result.blocked_duplicates == 1  # budget-limited, as expected
+        assert result.operational_expired == 1  # but age-out still progressed
+        assert result.carried_over is True
+        assert old_failed not in self.remaining_ids("st")
+
+    def test_delete_rechecks_survivor_after_a_pending_attempt_appears(self) -> None:
+        """Evaluate survivor safety inside the DELETE statement.
+
+        Constructing the candidate query must not freeze its result. A pending
+        attempt committed before execution can become a mid-history boundary,
+        so the later blocked row must remain available as its future survivor.
+        """
+        blocked_since: UUID = self.add_row(STATUS_BLOCKED, entity="rc", age_days=3)
+        later_block: UUID = self.add_row(STATUS_BLOCKED, entity="rc", age_days=1)
+        select_candidates: Callable[[int], sa.sql.Select] = partial(
+            prune_audit._duplicate_candidates, audit.utc_now()

Review Comment:
   This private-helper coupling is intentional in this regression. The test must construct the duplicate-candidate statement, commit a pending row after construction, and only then execute deletion; that controlled interleaving proves candidate safety is re-evaluated by the DELETE rather than frozen earlier. The public `run_prune()` entry point does not expose an insertion point for that invariant.



##########
tests/integration_tests/deletion_retention/prune_audit_tests.py:
##########
@@ -0,0 +1,484 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Integration tests for purge-audit pruning against the real metadata DB.
+
+Rows are seeded directly (pruning reads only ``purge_audit_log``); every
+test's rows carry a distinctive ``entity_type`` prefix and are removed in
+teardown, so no other suite ever sees them. Scoping on ``entity_type``
+rather than ``entity_uuid`` matters: some rows deliberately carry a NULL
+uuid, which a ``LIKE`` on the uuid column would never match.
+"""
+
+from __future__ import annotations
+
+from collections.abc import Callable
+from datetime import timedelta
+from functools import partial
+from typing import Any
+from unittest.mock import patch
+from uuid import UUID, uuid4
+
+import sqlalchemy as sa
+from flask import current_app
+from sqlalchemy.orm import Query
+
+from superset import db
+from superset.commands.deletion_retention import audit, prune_audit
+from superset.commands.deletion_retention.prune_audit import (
+    EVIDENCE_RETENTION_KEY,
+    OPERATIONAL_RETENTION_KEY,
+)
+from superset.models.purge_audit_log import (
+    PurgeAuditLog,
+    STATUS_BLOCKED,
+    STATUS_CONFIRMED,
+    STATUS_FAILED,
+    STATUS_PENDING,
+    STATUS_TARGET_ABSENT,
+)
+from tests.integration_tests.base_tests import SupersetTestCase
+from tests.integration_tests.deletion_retention._base import (
+    ensure_purge_audit_coordination,
+)
+
+_PREFIX: str = "prune_audit_it_"
+_ENTITY_TYPE: str = f"{_PREFIX}slices"
+
+
+class TestPruneAudit(SupersetTestCase):
+    """Behavioral coverage for ``prune_audit.run_prune``."""
+
+    _seq: int
+
+    def setUp(self) -> None:
+        super().setUp()
+        ensure_purge_audit_coordination()
+        self._seq = 0
+        self._cleanup()
+
+    def tearDown(self) -> None:
+        self._cleanup()
+        super().tearDown()
+
+    def _cleanup(self) -> None:
+        db.session.rollback()
+        db.session.execute(
+            sa.delete(PurgeAuditLog.__table__).where(
+                PurgeAuditLog.__table__.c.entity_type == _ENTITY_TYPE
+            )
+        )
+        db.session.commit()
+
+    def add_row(
+        self,
+        status: str,
+        entity: str | None = "e1",
+        age_days: float = 0,
+        trigger: str = audit.TRIGGER_RETENTION,
+    ) -> UUID:
+        """Seed one audit row and return its id.
+
+        Ids rather than instances: an instance whose row a later prune
+        deletes raises ObjectDeletedError on attribute access. ``age_days``
+        may be fractional; a strictly increasing microsecond sequence keeps
+        every ``created_on`` unique so streak ordering is deterministic.
+        ``entity=None`` seeds a row with no ``entity_uuid``.
+        """
+        self._seq += 1
+        row: PurgeAuditLog = PurgeAuditLog(
+            id=uuid4(),
+            status=status,
+            trigger=trigger,
+            actor=audit.ACTOR_SYSTEM,
+            entity_type=_ENTITY_TYPE,
+            entity_uuid=None if entity is None else f"{_PREFIX}{entity}",
+            created_on=audit.utc_now()
+            - timedelta(days=age_days)
+            + timedelta(microseconds=self._seq),
+        )
+        db.session.add(row)
+        db.session.commit()
+        return row.id
+
+    def remaining_ids(self, entity: str | None = "__any__") -> list[UUID]:
+        """Ids of surviving seeded rows, oldest first."""
+        query: Query[PurgeAuditLog] = db.session.query(PurgeAuditLog).filter(
+            PurgeAuditLog.entity_type == _ENTITY_TYPE
+        )
+        if entity != "__any__":
+            query = query.filter(
+                PurgeAuditLog.entity_uuid.is_(None)
+                if entity is None
+                else PurgeAuditLog.entity_uuid == f"{_PREFIX}{entity}"
+            )
+        return [r.id for r in query.order_by(PurgeAuditLog.created_on).all()]
+
+    def run_prune(self, **config: Any) -> prune_audit.PruneRunResult:
+        with patch.dict(current_app.config, config):
+            return prune_audit.run_prune()
+
+    # -- US1: bounded blockage history -------------------------------------
+
+    def test_duplicates_reduce_to_the_earliest_survivor_regardless_of_age(self) -> None:
+        """Keep only the earliest row in a current blockage streak."""
+        ids: list[UUID] = [
+            self.add_row(STATUS_BLOCKED, age_days=5 - i / 100) for i in range(50)
+        ]
+
+        result: prune_audit.PruneRunResult = self.run_prune()
+
+        assert result.blocked_duplicates == 49
+        assert result.carried_over is False
+        assert self.remaining_ids("e1") == [ids[0]]
+
+    def test_backlog_converges_over_bounded_runs(self) -> None:
+        """Drain oversized backlogs across bounded successive runs."""
+        for i in range(56):
+            self.add_row(STATUS_BLOCKED, age_days=3 - i / 1000)
+
+        removed_per_run: list[int] = []
+        with (
+            patch.object(prune_audit, "BATCH_SIZE", 10),
+            patch.object(prune_audit, "MAX_BATCHES_PER_RUN", 2),
+        ):
+            while True:
+                result: prune_audit.PruneRunResult = self.run_prune()
+                removed_per_run.append(result.blocked_duplicates)
+                if not result.carried_over:
+                    break
+
+        assert sum(removed_per_run) == 55  # every duplicate, none of the survivor
+        assert all(n <= 20 for n in removed_per_run)  # never exceeds the budget
+        assert removed_per_run[-1] > 0  # carryover is not reported spuriously
+        assert len(self.remaining_ids("e1")) == 1  # the survivor
+
+    def test_dedup_never_crosses_entities(self) -> None:
+        a_ids: list[UUID] = [
+            self.add_row(STATUS_BLOCKED, entity="a", age_days=2) for _ in range(3)
+        ]
+        b_ids: list[UUID] = [
+            self.add_row(STATUS_BLOCKED, entity="b", age_days=2) for _ in range(4)
+        ]
+
+        result: prune_audit.PruneRunResult = self.run_prune()
+
+        assert result.blocked_duplicates == 5
+        assert self.remaining_ids("a") == [a_ids[0]]
+        assert self.remaining_ids("b") == [b_ids[0]]
+
+    def test_resolved_streak_has_no_survivor_and_ages_out(self) -> None:
+        """Expire blocked rows after newer evidence resolves their streak."""
+        for _ in range(3):
+            self.add_row(STATUS_BLOCKED, age_days=200)
+        confirmed: UUID = self.add_row(STATUS_CONFIRMED, age_days=100)
+        new_survivor: UUID = self.add_row(STATUS_BLOCKED, age_days=50)
+        self.add_row(STATUS_BLOCKED, age_days=40)  # duplicate in the new streak
+
+        result: prune_audit.PruneRunResult = self.run_prune()
+
+        assert result.blocked_duplicates == 1
+        assert result.operational_expired == 3
+        assert set(self.remaining_ids("e1")) == {confirmed, new_survivor}
+
+    def test_a_failed_attempt_does_not_reset_the_blocked_since_survivor(self) -> None:
+        """Keep one blockage streak across a failed purge attempt."""
+        blocked_since: UUID = self.add_row(STATUS_BLOCKED, age_days=200)
+        self.add_row(STATUS_FAILED, age_days=150)  # ages out as operational
+        self.add_row(STATUS_BLOCKED, age_days=100)
+        self.add_row(STATUS_BLOCKED, age_days=50)
+
+        result: prune_audit.PruneRunResult = self.run_prune()
+
+        assert result.blocked_duplicates == 2
+        assert result.operational_expired == 1
+        assert self.remaining_ids("e1") == [blocked_since]
+
+    # -- US2: evidence survives by default ----------------------------------
+
+    def test_defaults_leave_completed_destruction_evidence_untouched(self) -> None:
+        evidence: list[UUID] = [
+            self.add_row(STATUS_CONFIRMED, entity="ev", age_days=3650),
+            self.add_row(STATUS_TARGET_ABSENT, entity="ev", age_days=1000),
+            self.add_row(STATUS_CONFIRMED, entity="ev", age_days=1),
+        ]
+        self.add_row(STATUS_FAILED, entity="ev", age_days=365)  # ages out
+
+        for _ in range(3):  # any sequence of runs (SC-002)
+            result: prune_audit.PruneRunResult = self.run_prune()
+
+        assert result.evidence_expired == 0
+        assert set(self.remaining_ids("ev")) == set(evidence)
+
+    def test_evidence_opt_in_expires_only_rows_older_than_its_window(self) -> None:
+        old: UUID = self.add_row(STATUS_CONFIRMED, entity="ev", age_days=400)
+        young: UUID = self.add_row(STATUS_TARGET_ABSENT, entity="ev", age_days=300)
+
+        result: prune_audit.PruneRunResult = self.run_prune(
+            **{EVIDENCE_RETENTION_KEY: 365}
+        )
+
+        assert result.evidence_expired == 1
+        assert self.remaining_ids("ev") == [young]
+        assert old not in self.remaining_ids("ev")
+
+    def test_opt_in_disabled_again_removes_no_further_evidence(self) -> None:
+        self.add_row(STATUS_CONFIRMED, entity="ev", age_days=400)
+        survivor: UUID = self.add_row(STATUS_CONFIRMED, entity="ev", age_days=390)
+
+        first: prune_audit.PruneRunResult = self.run_prune(
+            **{EVIDENCE_RETENTION_KEY: 395}
+        )
+        assert first.evidence_expired == 1
+
+        second: prune_audit.PruneRunResult = self.run_prune(
+            **{EVIDENCE_RETENTION_KEY: None}
+        )
+        assert second.evidence_expired == 0
+        assert self.remaining_ids("ev") == [survivor]
+
+    def test_pending_and_future_rows_survive_every_configuration(self) -> None:
+        pending: UUID = self.add_row(STATUS_PENDING, entity="px", age_days=3650)
+        future_blocked: UUID = self.add_row(STATUS_BLOCKED, entity="px", age_days=-1)
+        future_confirmed: UUID = self.add_row(
+            STATUS_CONFIRMED, entity="px", age_days=-2
+        )
+
+        result: prune_audit.PruneRunResult = self.run_prune(
+            **{OPERATIONAL_RETENTION_KEY: 1, EVIDENCE_RETENTION_KEY: 1}
+        )
+
+        assert result.total_removed == 0
+        assert set(self.remaining_ids("px")) == {
+            pending,
+            future_blocked,
+            future_confirmed,
+        }
+
+    def test_a_future_dated_outcome_cannot_resolve_a_live_streak(self) -> None:
+        """Exclude future-dated evidence from current streak boundaries."""
+        blocked_since: UUID = self.add_row(STATUS_BLOCKED, entity="sk", age_days=500)
+        self.add_row(STATUS_BLOCKED, entity="sk", age_days=400)
+        skewed: UUID = self.add_row(STATUS_CONFIRMED, entity="sk", age_days=-30)
+
+        result: prune_audit.PruneRunResult = self.run_prune()
+
+        assert result.blocked_duplicates == 1
+        assert result.operational_expired == 0
+        assert set(self.remaining_ids("sk")) == {blocked_since, skewed}
+
+    # -- Survivor invariant under boundary removal (the review's HIGH) ------
+
+    def test_evidence_expiry_spares_a_boundary_that_still_bounds_blocked_rows(
+        self,
+    ) -> None:
+        """Keep evidence while it still bounds surviving blocked rows."""
+        blocked: list[UUID] = [
+            self.add_row(STATUS_BLOCKED, entity="bd", age_days=30) for _ in range(3)
+        ]
+        boundary: UUID = self.add_row(STATUS_CONFIRMED, entity="bd", age_days=20)
+
+        # Evidence window far shorter than the operational one: the boundary
+        # is past its cutoff, the blocked rows it resolved are not.
+        first: prune_audit.PruneRunResult = self.run_prune(
+            **{OPERATIONAL_RETENTION_KEY: 90, EVIDENCE_RETENTION_KEY: 1}
+        )
+
+        assert first.evidence_expired == 0
+        assert set(self.remaining_ids("bd")) == {*blocked, boundary}
+
+        # Once the blocked rows age out, the boundary bounds nothing and
+        # becomes expirable — the guard defers, it does not immortalize.
+        # Categories drain in order within one run, so the blocked rows go
+        # first and the boundary follows in the same pass.
+        second: prune_audit.PruneRunResult = self.run_prune(
+            **{OPERATIONAL_RETENTION_KEY: 10, EVIDENCE_RETENTION_KEY: 1}
+        )
+        assert second.operational_expired == 3
+        assert second.evidence_expired == 1
+        assert self.remaining_ids("bd") == []
+
+    def test_a_block_after_an_unresolved_attempt_is_not_treated_as_a_duplicate(
+        self,
+    ) -> None:
+        """An in-flight attempt is a boundary waiting to happen.
+
+        Finalizing a ``pending`` row resolves it *in place*, keeping its
+        original timestamp — the one way a boundary can appear in the middle
+        of history. The blocked row after it would become the new streak's
+        survivor, so it must not be classified as a duplicate while the
+        attempt is unresolved. Deleting it would destroy the only record
+        that the entity was still blocked after that attempt.
+        """
+        blocked_since: UUID = self.add_row(STATUS_BLOCKED, entity="ua", age_days=100)
+        attempt: UUID = self.add_row(STATUS_PENDING, entity="ua", age_days=50)
+        later_block: UUID = self.add_row(STATUS_BLOCKED, entity="ua", age_days=10)
+
+        result: prune_audit.PruneRunResult = self.run_prune()
+
+        assert result.blocked_duplicates == 0
+        assert set(self.remaining_ids("ua")) == {blocked_since, attempt, later_block}
+
+        # Once the attempt finalizes, the boundary is real: the later block
+        # is the new streak's survivor and stays; the older block is now a
+        # resolved-streak row that ages out on the normal window.
+        db.session.execute(
+            sa.update(PurgeAuditLog.__table__)
+            .where(PurgeAuditLog.__table__.c.id == attempt)
+            .values(status=STATUS_CONFIRMED)
+        )
+        db.session.commit()
+
+        after: prune_audit.PruneRunResult = self.run_prune()
+
+        assert after.blocked_duplicates == 0
+        assert after.operational_expired == 1  # the pre-attempt block
+        assert set(self.remaining_ids("ua")) == {attempt, later_block}
+
+    def test_age_does_not_make_an_unstable_block_expirable(self) -> None:
+        """Keep aged blocked rows whose classification remains unstable.
+
+        Seeded entirely outside the retention window, so only the guard —
+        not the cutoff — can save the row.
+        """
+        blocked_since: UUID = self.add_row(STATUS_BLOCKED, entity="ao", age_days=100)
+        attempt: UUID = self.add_row(STATUS_PENDING, entity="ao", age_days=98)
+        later_block: UUID = self.add_row(STATUS_BLOCKED, entity="ao", age_days=95)
+
+        result: prune_audit.PruneRunResult = self.run_prune()
+
+        assert result.blocked_duplicates == 0
+        assert result.operational_expired == 0
+        assert set(self.remaining_ids("ao")) == {blocked_since, attempt, later_block}
+
+        # Resolving the attempt makes the boundary real: the later block is
+        # the current streak's survivor and is exempt regardless of age,
+        # while the pre-attempt block ages out.
+        db.session.execute(
+            sa.update(PurgeAuditLog.__table__)
+            .where(PurgeAuditLog.__table__.c.id == attempt)
+            .values(status=STATUS_TARGET_ABSENT)
+        )
+        db.session.commit()
+
+        after: prune_audit.PruneRunResult = self.run_prune()
+
+        assert after.operational_expired == 1
+        assert set(self.remaining_ids("ao")) == {attempt, later_block}
+
+    def test_evidence_guard_also_defers_to_an_unresolved_older_attempt(self) -> None:
+        """Treat an older pending attempt as potential blocked evidence."""
+        attempt: UUID = self.add_row(STATUS_PENDING, entity="ug", age_days=400)
+        boundary: UUID = self.add_row(STATUS_CONFIRMED, entity="ug", age_days=300)
+
+        result: prune_audit.PruneRunResult = self.run_prune(
+            **{EVIDENCE_RETENTION_KEY: 200}
+        )
+
+        assert result.evidence_expired == 0
+        assert set(self.remaining_ids("ug")) == {attempt, boundary}
+
+    def test_blocked_rows_without_an_entity_uuid_are_never_aged_out(self) -> None:
+        """Keep UUID-less blocked rows that cannot be proven redundant."""
+        anonymous_block: UUID = self.add_row(STATUS_BLOCKED, entity=None, age_days=1000)
+        self.add_row(STATUS_FAILED, entity=None, age_days=1000)
+
+        result: prune_audit.PruneRunResult = self.run_prune()
+
+        assert result.operational_expired == 1
+        assert self.remaining_ids(None) == [anonymous_block]
+
+    # -- US3: operator controls and observability ---------------------------
+
+    def test_configured_retention_window_is_honored(self) -> None:
+        kept: UUID = self.add_row(STATUS_FAILED, entity="w", age_days=5)
+        self.add_row(STATUS_FAILED, entity="w", age_days=15)
+
+        result: prune_audit.PruneRunResult = self.run_prune(
+            **{OPERATIONAL_RETENTION_KEY: 10}
+        )
+
+        assert result.operational_expired == 1
+        assert self.remaining_ids("w") == [kept]
+
+    def test_invalid_operational_window_skips_the_category_not_widens(self) -> None:
+        """Fail closed when the operational retention window is invalid."""
+        old_failed: UUID = self.add_row(STATUS_FAILED, entity="iv", age_days=1000)
+        self.add_row(STATUS_BLOCKED, entity="iv", age_days=3)
+        self.add_row(STATUS_BLOCKED, entity="iv", age_days=2)
+
+        result: prune_audit.PruneRunResult = self.run_prune(
+            **{OPERATIONAL_RETENTION_KEY: 0}
+        )
+
+        assert result.operational_expired == 0
+        assert result.blocked_duplicates == 1
+        assert result.invalid_config_keys == [OPERATIONAL_RETENTION_KEY]
+        assert old_failed in self.remaining_ids("iv")
+
+    def test_second_run_over_the_same_candidates_removes_and_reports_zero(self) -> None:
+        """Report zero when rerunning over an already-pruned history."""
+        for _ in range(4):
+            self.add_row(STATUS_BLOCKED, entity="cc", age_days=2)
+        self.add_row(STATUS_FAILED, entity="cc", age_days=100)
+
+        first: prune_audit.PruneRunResult = self.run_prune()
+        assert first.blocked_duplicates == 3
+        assert first.operational_expired == 1
+
+        second: prune_audit.PruneRunResult = self.run_prune()
+        assert second.total_removed == 0
+        assert len(self.remaining_ids("cc")) == 1
+
+    def test_a_duplicate_backlog_cannot_starve_the_age_based_categories(self) -> None:
+        """Prevent duplicate backlogs from starving age-based categories."""
+        for _ in range(20):
+            self.add_row(STATUS_BLOCKED, entity="st", age_days=2)
+        old_failed: UUID = self.add_row(STATUS_FAILED, entity="st", age_days=1000)
+
+        with (
+            patch.object(prune_audit, "BATCH_SIZE", 1),
+            patch.object(prune_audit, "MAX_BATCHES_PER_RUN", 2),
+        ):
+            result: prune_audit.PruneRunResult = self.run_prune()
+
+        assert result.blocked_duplicates == 1  # budget-limited, as expected
+        assert result.operational_expired == 1  # but age-out still progressed
+        assert result.carried_over is True
+        assert old_failed not in self.remaining_ids("st")
+
+    def test_delete_rechecks_survivor_after_a_pending_attempt_appears(self) -> None:
+        """Evaluate survivor safety inside the DELETE statement.
+
+        Constructing the candidate query must not freeze its result. A pending
+        attempt committed before execution can become a mid-history boundary,
+        so the later blocked row must remain available as its future survivor.
+        """
+        blocked_since: UUID = self.add_row(STATUS_BLOCKED, entity="rc", age_days=3)
+        later_block: UUID = self.add_row(STATUS_BLOCKED, entity="rc", age_days=1)
+        select_candidates: Callable[[int], sa.sql.Select] = partial(
+            prune_audit._duplicate_candidates, audit.utc_now()
+        )
+
+        attempt: UUID = self.add_row(STATUS_PENDING, entity="rc", age_days=2)
+        removed: int = prune_audit._delete_batch(select_candidates)

Review Comment:
   This call is paired with the intentionally constructed candidate selector above. Calling `_delete_batch` directly is what lets the regression execute exactly one deletion after the pending row is committed and assert the atomic re-evaluation contract. Going through `run_prune()` would lose control of that ordering and make the test unable to distinguish the original race.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org


Re: [PR] feat(deletion-retention): scheduled pruning policy for the purge audit log [superset]

Posted by "bito-code-review[bot] (via GitHub)" <gi...@apache.org>.
bito-code-review[bot] commented on PR #43490:
URL: https://github.com/apache/superset/pull/43490#issuecomment-5418309380

   AI Code Review is in progress (usually takes 3 to 15 minutes unless it's a very large PR).
   
   <details>
   <summary>
   Bito Usage Guide
   </summary>
   
   **Commands**
         
   Type the following command in the pull request comment and save the comment.
         
   - `/review` - Manually triggers an incremental AI Review.
   - `/review full` - Manually triggers a full AI Review.
   
   - `/pause` - Pauses automatic reviews on this pull request.
   - `/resume` - Resumes automatic reviews.
   - `/resolve` - Marks all Bito-posted review comments as resolved.
   - `/abort` - Cancels all in-progress reviews.
   
         
   Refer to the <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/available-commands" target="_blank" rel="noopener noreferrer">documentation</a> for additional commands.
         
   **Configuration**
   
   This repository uses `Superset` You can customize the agent settings <a href="https://alpha.bito.ai/home/ai-agents/code-review-agent" target="_blank" rel="noopener noreferrer">here</a> or contact your Bito workspace admin at evan@preset.io.
         
   **Documentation & Help**
   - <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/getting-started/install-run-using-bito-cloud/create-or-customize-an-agent-instance" target="_blank" rel="noopener noreferrer">Customize agent settings</a>
   - <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/implementing-custom-code-review-rules" target="_blank" rel="noopener noreferrer">Review rules</a>
   - <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent" target="_blank" rel="noopener noreferrer">General documentation</a>
   - <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/faqs" target="_blank" rel="noopener noreferrer">FAQ</a>
   
   </details>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org


Re: [PR] feat(deletion-retention): scheduled pruning policy for the purge audit log [superset]

Posted by "codeant-ai-for-open-source[bot] (via GitHub)" <gi...@apache.org>.
codeant-ai-for-open-source[bot] commented on code in PR #43490:
URL: https://github.com/apache/superset/pull/43490#discussion_r3864763006


##########
superset/initialization/__init__.py:
##########
@@ -1002,11 +1005,14 @@ def _warn_if_retention_beat_missing(self) -> None:
                 "CeleryConfig or add the module to your override.",
                 self._RETENTION_TASK_MODULE,
             )
-        default_flags = self.config.get("DEFAULT_FEATURE_FLAGS", {})
-        configured_flags = self.config.get("FEATURE_FLAGS", {})
-        soft_delete_enabled = bool(
+        default_flags: dict[str, Any] = self.config.get("DEFAULT_FEATURE_FLAGS", {})
+        configured_flags: dict[str, Any] = self.config.get("FEATURE_FLAGS", {})
+        soft_delete_enabled: bool = bool(
             configured_flags.get("SOFT_DELETE", default_flags.get("SOFT_DELETE", False))
         )
+        audit_pruning_enabled: bool = (
+            self.config.get("PURGE_AUDIT_PRUNING_ENABLED", False) is True
+        )

Review Comment:
   The comment is valid. `is True` correctly prevents invalid values from enabling pruning, but it also collapses invalid values into the legitimate `False` state, so startup provides no actionable warning.
   
   Keep the fail-closed behavior, but validate the type separately:
   
   ```python
   audit_pruning_config: Any = self.config.get(
       "PURGE_AUDIT_PRUNING_ENABLED", False
   )
   if not isinstance(audit_pruning_config, bool):
       logger.warning(
           "deletion-retention: invalid PURGE_AUDIT_PRUNING_ENABLED=%r; "
           "audit pruning remains disabled",
           audit_pruning_config,
       )
       audit_pruning_enabled = False
   else:
       audit_pruning_enabled = audit_pruning_config
   ```
   
   Then retain the existing schedule/import checks based on `audit_pruning_enabled`. This preserves the task’s strict behavior while making values such as `"true"` and `1` visible during initialization. A focused test should assert that an invalid value logs a warning and does not trigger the schedule check.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org


Re: [PR] feat(deletion-retention): scheduled pruning policy for the purge audit log [superset]

Posted by "bito-code-review[bot] (via GitHub)" <gi...@apache.org>.
bito-code-review[bot] commented on PR #43490:
URL: https://github.com/apache/superset/pull/43490#issuecomment-5614092934

   AI Code Review is in progress (usually takes 3 to 15 minutes unless it's a very large PR).
   
   <details>
   <summary>
   Bito Usage Guide
   </summary>
   
   **Commands**
         
   Type the following command in the pull request comment and save the comment.
         
   - `/review` - Manually triggers an incremental AI Review.
   - `/review full` - Manually triggers a full AI Review.
   
   - `/pause` - Pauses automatic reviews on this pull request.
   - `/resume` - Resumes automatic reviews.
   - `/resolve` - Marks all Bito-posted review comments as resolved.
   - `/abort` - Cancels all in-progress reviews.
   
         
   Refer to the <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/available-commands" target="_blank" rel="noopener noreferrer">documentation</a> for additional commands.
         
   **Configuration**
   
   This repository uses `Superset` You can customize the agent settings <a href="https://alpha.bito.ai/home/ai-agents/code-review-agent" target="_blank" rel="noopener noreferrer">here</a> or contact your Bito workspace admin at evan@preset.io.
         
   **Documentation & Help**
   - <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/getting-started/install-run-using-bito-cloud/create-or-customize-an-agent-instance" target="_blank" rel="noopener noreferrer">Customize agent settings</a>
   - <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/implementing-custom-code-review-rules" target="_blank" rel="noopener noreferrer">Review rules</a>
   - <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent" target="_blank" rel="noopener noreferrer">General documentation</a>
   - <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/faqs" target="_blank" rel="noopener noreferrer">FAQ</a>
   
   </details>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org


Re: [PR] feat(deletion-retention): scheduled pruning policy for the purge audit log [superset]

Posted by "mikebridge (via GitHub)" <gi...@apache.org>.
mikebridge commented on code in PR #43490:
URL: https://github.com/apache/superset/pull/43490#discussion_r3931385349


##########
superset/commands/deletion_retention/prune_audit.py:
##########
@@ -0,0 +1,607 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Deletion-only pruning of the ``purge_audit_log`` table.
+
+Bounds the audit history's growth without ever weakening its evidentiary
+value. Three delete categories, applied in priority order under one shared
+per-run batch budget:
+
+1. **Blocked duplicates** — within an entity's *current* blockage streak
+   (its ``blocked`` rows newer than the entity's newest streak-breaking
+   row), only the streak's earliest row survives: later duplicates are
+   removed regardless of age. The survivor carries the "blocked since"
+   fact and is never deleted while the streak is current.
+2. **Operational expiry** — ``blocked`` rows of *resolved* streaks and
+   ``failed`` rows older than ``PURGE_AUDIT_OPERATIONAL_RETENTION_DAYS``
+   age out.
+3. **Evidence expiry** — ``confirmed`` / ``target_absent`` rows are
+   untouchable unless ``PURGE_AUDIT_EVIDENCE_RETENTION_DAYS`` is
+   explicitly set (the operator's compliance assertion), and then only
+   rows older than that window.
+
+``pending`` rows belong to :func:`audit.reconcile_pending` and rows with
+future timestamps (clock skew) are excluded from every category *and* from
+streak classification, so a skewed writer cannot reclassify a live streak.
+
+Three invariants keep the survivor safe without a distributed lock, which
+matters because runs can overlap and both ``reconcile_pending`` and the
+purge path finalize rows concurrently. A duplicate is only ever deleted
+when no concurrent transition could turn it into a survivor first:
+
+* **A boundary is never removed while it still bounds anything.** Evidence
+  expiry refuses to delete a row while an older ``blocked`` or ``pending``
+  row for the same entity survives. Boundaries therefore never *recede*,
+  which would otherwise promote resolved rows into a current streak.
+* **A blocked row whose classification is unstable is never deleted.**
+  Finalizing a ``pending`` row resolves it *in place*, keeping its original
+  timestamp, so an unresolved attempt is a boundary that may appear
+  mid-history at any moment — and the blocked row after it would become its
+  streak's survivor. Blocked rows preceded by an unresolved attempt are
+  therefore skipped by **both** the duplicate and the operational category.
+  Age does not stabilize the classification: an old blocked row inside a
+  live streak is exactly the "blocked for years" case FR-009 protects.
+  Without this, a boundary moving *forward* would demote the current
+  survivor and promote the next row into its place — possibly a row already
+  selected for deletion.
+* **Deletes are conditional and counted from rowcounts.** A row whose
+  status changed since selection is not matched, so overlapping runs can
+  neither double-remove nor double-report.
+
+Boundaries moving forward past *all* of an entity's blocked rows leave no
+survivor to promote, so a selected duplicate may be removed slightly ahead
+of its retention window in that case. It was redundant either way and the
+streak's earliest row is untouched.
+
+Candidate selection is embedded in each ``DELETE`` statement. The derived-table
+wrapper keeps that shape legal on MySQL while ensuring that a pending or
+recovered row committed before the delete is evaluated participates in the
+survivor and boundary predicates. No stale list of candidate ids crosses a
+transaction boundary.
+
+Audit creation/recovery and every pruning batch take the same singleton
+database write lock before assigning a timestamp or evaluating candidates.
+The lock is held through commit, so an audit row cannot become visible in an
+already-processed logical past. Automatic pruning still ships disabled by
+default so operators explicitly choose their retention policy.
+"""
+
+from __future__ import annotations
+
+import logging
+from collections.abc import Callable
+from dataclasses import dataclass, field
+from datetime import datetime, timedelta
+from functools import partial
+from typing import Any, Literal, NamedTuple, TypeAlias
+
+import sqlalchemy as sa
+from flask import current_app
+
+from superset import db
+from superset.commands.deletion_retention.audit import (
+    acquire_coordination_lock,
+    utc_now,
+)
+from superset.models.purge_audit_log import (
+    PurgeAuditLog,
+    STATUS_BLOCKED,
+    STATUS_CONFIRMED,
+    STATUS_FAILED,
+    STATUS_PENDING,
+    STATUS_TARGET_ABSENT,
+)
+
+logger: logging.Logger = logging.getLogger(__name__)
+
+#: Operational records: noise-prone outcomes whose compliance value decays —
+#: a blocked or failed purge leaves the object in place (FR-001).
+OPERATIONAL_STATUSES: frozenset[str] = frozenset({STATUS_BLOCKED, STATUS_FAILED})
+#: Protected evidence: the only surviving trace of a destroyed object.
+PROTECTED_STATUSES: frozenset[str] = frozenset({STATUS_CONFIRMED, STATUS_TARGET_ABSENT})
+#: The outcomes that end a blockage streak — proof the object is gone.
+#:
+#: ``failed`` is deliberately absent. A failed purge is an infrastructure
+#: outcome (the cascade raised), not evidence the blockage cleared: the
+#: policy that blocked the entity is untouched, so the blockage continues
+#: across it. Treating ``failed`` as a boundary would let one transient
+#: error demote the "blocked since" survivor to an ageing duplicate and
+#: restate the blockage as beginning after the failure — losing exactly the
+#: fact FR-003 exists to preserve. ``pending`` is provisional and likewise
+#: neither joins nor breaks streaks.
+_STREAK_BREAKING_STATUSES: frozenset[str] = frozenset(
+    {STATUS_CONFIRMED, STATUS_TARGET_ABSENT}
+)
+
+#: Rows deleted per statement, matching the purge task's batch convention.
+BATCH_SIZE: int = 500
+#: One shared budget for the whole run across all three categories; the
+#: remaining backlog carries over to the next scheduled run (FR-004/SC-004).
+MAX_BATCHES_PER_RUN: int = 10
+
+OPERATIONAL_RETENTION_KEY: str = "PURGE_AUDIT_OPERATIONAL_RETENTION_DAYS"
+EVIDENCE_RETENTION_KEY: str = "PURGE_AUDIT_EVIDENCE_RETENTION_DAYS"
+
+
+class ResolvedWindow(NamedTuple):
+    """A retention window resolved from config.
+
+    Distinguishes the three outcomes a caller must tell apart: a usable
+    window, the deliberate "off" default, and operator error. Collapsing
+    the last two into a bare ``None`` would force every caller to re-read
+    config to find out which it got.
+    """
+
+    days: int | None
+    invalid_key: str | None = None
+
+
+def _validated_window(key: str, value: Any) -> ResolvedWindow:
+    """Validate a configured day count, failing closed on anything odd.
+
+    An invalid value disables its category for the run rather than widening
+    removal (FR-005/SC-005).
+    """
+    # bool is an int subclass and floats would silently truncate — both are
+    # config mistakes, not day counts, on a knob that deletes rows.
+    if not isinstance(value, bool) and not isinstance(value, float):
+        try:
+            days: int = int(value)
+        except (TypeError, ValueError):
+            days = 0
+        if days > 0:
+            return ResolvedWindow(days)
+    logger.warning(
+        "prune_audit: invalid %s=%r; skipping this category for the run "
+        "(pruning never widens on bad configuration)",
+        key,
+        value,
+    )
+    return ResolvedWindow(None, key)
+
+
+def resolve_operational_retention_days() -> ResolvedWindow:
+    """The operational retention window, or a disabled window when invalid."""
+    return _validated_window(
+        OPERATIONAL_RETENTION_KEY, current_app.config.get(OPERATIONAL_RETENTION_KEY)
+    )
+
+
+def resolve_evidence_retention_days() -> ResolvedWindow:
+    """The evidence expiration window; disabled unless explicitly opted in.
+
+    Unset is the documented "never expire evidence" default (FR-006), not an
+    error, so it produces a disabled window with no warning.
+    """
+    value: Any = current_app.config.get(EVIDENCE_RETENTION_KEY)
+    if value is None:
+        return ResolvedWindow(None)
+    return _validated_window(EVIDENCE_RETENTION_KEY, value)
+
+
+@dataclass
+class PruneRunResult:
+    """Per-category removal counts and run disposition for one pruning run."""
+
+    blocked_duplicates: int = 0
+    operational_expired: int = 0
+    evidence_expired: int = 0
+    #: True when the shared batch budget ran out before every category's
+    #: candidates were drained; the remainder converges on later runs.
+    carried_over: bool = False
+    invalid_config_keys: list[str] = field(default_factory=list)
+
+    @property
+    def total_removed(self) -> int:
+        """Total rows removed across every category."""
+        return (
+            self.blocked_duplicates + self.operational_expired + self.evidence_expired
+        )
+
+    def as_dict(self) -> dict[str, Any]:
+        """The task-return / log-line shape of this result."""
+        return {
+            "removed": {
+                "blocked_duplicates": self.blocked_duplicates,
+                "operational_expired": self.operational_expired,
+                "evidence_expired": self.evidence_expired,
+            },
+            "carried_over": self.carried_over,
+            "invalid_config_keys": list(self.invalid_config_keys),
+        }
+
+
+def _streak_boundary_subquery(now: datetime) -> sa.Subquery:
+    """Per entity, the ``created_on`` of its newest streak-breaking row.
+
+    Entities absent from this subquery have never been proven destroyed —
+    all their blocked rows form one current streak. Future-dated rows are
+    excluded so a skewed writer clock cannot push the boundary ahead of a
+    live streak and make its rows look resolved.
+    """
+    table: sa.Table = PurgeAuditLog.__table__
+    return (
+        sa.select(
+            table.c.entity_type.label("entity_type"),
+            table.c.entity_uuid.label("entity_uuid"),
+            sa.func.max(table.c.created_on).label("boundary"),
+        )
+        .where(table.c.status.in_(_STREAK_BREAKING_STATUSES))
+        .where(table.c.entity_uuid.is_not(None))
+        .where(table.c.created_on <= now)
+        .group_by(table.c.entity_type, table.c.entity_uuid)
+        .subquery("streak_boundary")
+    )
+
+
+def _survivor_subquery(now: datetime) -> sa.Subquery:
+    """Per entity, the ``created_on`` of its current streak's earliest row.
+
+    The survivor is the earliest ``blocked`` row strictly newer than the
+    entity's boundary (or its earliest blocked row outright when no boundary
+    exists). Rows sharing that exact timestamp are all treated as survivors —
+    with microsecond precision ties are pathological, and keeping an extra
+    row errs on the preserving side.
+    """
+    table: sa.Table = PurgeAuditLog.__table__
+    boundary: sa.Subquery = _streak_boundary_subquery(now)
+    return (
+        sa.select(
+            table.c.entity_type.label("entity_type"),
+            table.c.entity_uuid.label("entity_uuid"),
+            sa.func.min(table.c.created_on).label("survivor_created_on"),
+        )
+        .select_from(
+            table.outerjoin(
+                boundary,
+                sa.and_(
+                    table.c.entity_type == boundary.c.entity_type,
+                    table.c.entity_uuid == boundary.c.entity_uuid,
+                ),
+            )
+        )
+        .where(table.c.status == STATUS_BLOCKED)
+        .where(table.c.entity_uuid.is_not(None))
+        .where(table.c.created_on <= now)
+        .where(
+            sa.or_(
+                boundary.c.boundary.is_(None),
+                table.c.created_on > boundary.c.boundary,
+            )
+        )
+        .group_by(table.c.entity_type, table.c.entity_uuid)
+        .subquery("streak_survivor")
+    )
+
+
+def _preceded_by_unresolved_attempt(table: sa.Table) -> sa.ColumnElement[bool]:
+    """Whether an unresolved (``pending``) attempt precedes this row.
+
+    A ``pending`` row is the only thing that can insert a streak boundary
+    into *history*: every other write lands at ``now``, newer than every
+    existing row, whereas reconciliation and the purge path finalize a
+    pending row **in place**, keeping its original ``created_on``. So a
+    pending row sitting between two blocked rows is a boundary that may
+    appear at any moment, and the blocked row after it would become the new
+    streak's survivor — the very row pruning must never delete.
+    """
+    pending: sa.Table = table.alias("unresolved_attempt")
+    return sa.exists(
+        sa.select(sa.literal(1))
+        .select_from(pending)
+        .where(
+            sa.and_(
+                pending.c.status == STATUS_PENDING,
+                pending.c.entity_type == table.c.entity_type,
+                pending.c.entity_uuid == table.c.entity_uuid,
+                pending.c.created_on < table.c.created_on,
+            )
+        )
+    )
+
+
+def _duplicate_candidates(now: datetime, limit: int) -> sa.sql.Select:
+    """Select current-streak blocked rows that are not the streak survivor.
+
+    Age-independent by design (FR-003): a duplicate is prunable the moment a
+    streak has a survivor, regardless of the retention window. Trigger is not
+    a discriminator — ``scheduled`` and ``force`` blocked rows share streaks.
+
+    Rows preceded by an unresolved attempt are skipped: their classification
+    is not stable, because that attempt can finalize into a boundary and
+    promote them to survivor between selection and deletion. Such rows are
+    collected once the attempt resolves. Note that pruning does not depend on
+    that happening promptly — reconciliation runs from the purge task, which
+    a deployment may have disabled or left in dry-run — so a long-lived
+    pending row defers its successors indefinitely rather than risking them.
+    """
+    table: sa.Table = PurgeAuditLog.__table__
+    survivor: sa.Subquery = _survivor_subquery(now)
+    return (
+        sa.select(table.c.id)
+        .select_from(
+            table.join(
+                survivor,
+                sa.and_(
+                    table.c.entity_type == survivor.c.entity_type,
+                    table.c.entity_uuid == survivor.c.entity_uuid,
+                ),
+            )
+        )
+        .where(table.c.status == STATUS_BLOCKED)
+        .where(table.c.created_on > survivor.c.survivor_created_on)

Review Comment:
   Verified COVERED. The reason transition is preserved — a `report_schedule` (or any) block following a different-reason block is a run head and survives, even with a later same-reason repeat behind it.
   
   Built the scenario: `blocked(reason B) → blocked(reason A=report_schedule) → blocked(reason A)`. Result at head: the earliest row and the reason-transition row (A) both survive; only the trailing A repeat is pruned. The mechanism is in `_repeats_an_earlier_block`: a row is a repeat only if there is an earlier **same-reason** block (`is_not_distinct_from`) with **no differing-reason block between** them (`reason_changed_between`, inclusive bounds). The first block of reason A has no earlier reason-A block, so it is never a repeat.
   
   Evidence (`test_reason_transition_survives_amid_repeats`, in `ac9bb3bf34`):
   - Against head: passes (transition retained).
   - Reverted-fix control: drop the reason discrimination from `_repeats_an_earlier_block` and the test fails (the transition is pruned) — confirming the reason logic is what preserves it.
   
   Test added to lock it in.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org


Re: [PR] feat(deletion-retention): scheduled pruning policy for the purge audit log [superset]

Posted by "sadpandajoe (via GitHub)" <gi...@apache.org>.
sadpandajoe commented on code in PR #43490:
URL: https://github.com/apache/superset/pull/43490#discussion_r3931218802


##########
superset/commands/deletion_retention/prune_audit.py:
##########
@@ -0,0 +1,689 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Deletion-only pruning of the ``purge_audit_log`` table.
+
+Bounds the audit history's growth without ever weakening its evidentiary
+value. Three delete categories, applied in priority order under one shared
+per-run batch budget:
+
+1. **Blocked duplicates** — within an entity's *current* blockage streak
+   (its ``blocked`` rows newer than the entity's newest streak-breaking
+   row), only the first row of each run of consecutive same-reason rows
+   survives: the streak's earliest row and the first row after every
+   change of block reason. Later same-reason repeats are removed
+   regardless of age. The survivors carry the "blocked since" fact and
+   the reason history — for coded reasons, the same rows the audit writer's
+   own suppression rule retains (reason-less legacy runs are additionally
+   collapsed to their earliest here) — and are never deleted while the
+   streak is current.
+2. **Operational expiry** — ``blocked`` rows of *resolved* streaks and
+   ``failed`` rows older than ``PURGE_AUDIT_OPERATIONAL_RETENTION_DAYS``
+   age out.
+3. **Evidence expiry** — ``confirmed`` / ``target_absent`` rows are
+   untouchable unless ``PURGE_AUDIT_EVIDENCE_RETENTION_DAYS`` is
+   explicitly set (the operator's compliance assertion), and then only
+   rows older than that window.
+
+``pending`` rows belong to :func:`audit.reconcile_pending` and rows with
+future timestamps (clock skew) are excluded from every category *and* from
+streak classification, so a skewed writer cannot reclassify a live streak.
+
+Three invariants keep the survivor safe without a distributed lock, which
+matters because runs can overlap and both ``reconcile_pending`` and the
+purge path finalize rows concurrently. A duplicate is only ever deleted
+when no concurrent transition could turn it into a survivor first:
+
+* **A boundary is never removed while it still bounds anything.** Evidence
+  expiry refuses to delete a row while an older ``blocked`` or ``pending``
+  row for the same entity survives. Boundaries therefore never *recede*,
+  which would otherwise promote resolved rows into a current streak.
+* **A blocked row whose classification is unstable is never deleted.**
+  Finalizing a ``pending`` row resolves it *in place*, keeping its original
+  timestamp, so an unresolved attempt is a boundary that may appear
+  mid-history at any moment — and the blocked row after it would become its
+  streak's survivor. Blocked rows preceded by an unresolved attempt are
+  therefore skipped by **both** the duplicate and the operational category.
+  Age does not stabilize the classification: an old blocked row inside a
+  live streak is exactly the "blocked for years" case FR-009 protects.
+  Without this, a boundary moving *forward* would demote the current
+  survivor and promote the next row into its place — possibly a row already
+  selected for deletion.
+* **Deletes are conditional and counted from rowcounts.** A row whose
+  status changed since selection is not matched, so overlapping runs can
+  neither double-remove nor double-report.
+
+Boundaries moving forward past *all* of an entity's blocked rows leave no
+survivor to promote, so a selected duplicate may be removed slightly ahead
+of its retention window in that case. It was redundant either way and the
+streak's earliest row is untouched.
+
+Timestamps order rows. Ties are possible — legacy second-precision rows,
+two writers within one clock tick — and are resolved on the preserving
+side: a ``pending`` row tied with a blocked row counts as preceding it (the
+block is deferred until the attempt resolves); a blocked row tied with a
+boundary sits on the boundary's *resolved* side (it ages out instead of
+seeding a new current streak, and the boundary is not removed before it);
+and tied same-reason blocked rows are all retained.
+
+Candidate selection is embedded in each ``DELETE`` statement. The derived-table
+wrapper keeps that shape legal on MySQL while ensuring that a pending or
+recovered row committed before the delete is evaluated participates in the
+survivor and boundary predicates. No stale list of candidate ids crosses a
+transaction boundary.
+
+Audit creation/recovery and every pruning batch take the same singleton
+database write lock before assigning a timestamp or evaluating candidates.
+The lock is held through commit, so an audit row cannot become visible in an
+already-processed logical past. Automatic pruning still ships disabled by
+default so operators explicitly choose their retention policy.
+"""
+
+from __future__ import annotations
+
+import logging
+from collections.abc import Callable
+from dataclasses import dataclass, field
+from datetime import datetime, timedelta
+from functools import partial
+from typing import Any, Literal, NamedTuple, TypeAlias
+
+import sqlalchemy as sa
+from flask import current_app
+
+from superset import db
+from superset.commands.deletion_retention.audit import (
+    acquire_coordination_lock,
+    utc_now,
+)
+from superset.models.purge_audit_log import (
+    PurgeAuditLog,
+    STATUS_BLOCKED,
+    STATUS_CONFIRMED,
+    STATUS_FAILED,
+    STATUS_PENDING,
+    STATUS_TARGET_ABSENT,
+)
+
+logger: logging.Logger = logging.getLogger(__name__)
+
+#: Operational records: noise-prone outcomes whose compliance value decays —
+#: a blocked or failed purge leaves the object in place (FR-001).
+OPERATIONAL_STATUSES: frozenset[str] = frozenset({STATUS_BLOCKED, STATUS_FAILED})
+#: Protected evidence: the only surviving trace of a destroyed object.
+PROTECTED_STATUSES: frozenset[str] = frozenset({STATUS_CONFIRMED, STATUS_TARGET_ABSENT})
+#: The outcomes that end a blockage streak — proof the object is gone.
+#:
+#: ``failed`` is deliberately absent. A failed purge is an infrastructure
+#: outcome (the cascade raised), not evidence the blockage cleared: the
+#: policy that blocked the entity is untouched, so the blockage continues
+#: across it. Treating ``failed`` as a boundary would let one transient
+#: error demote the "blocked since" survivor to an ageing duplicate and
+#: restate the blockage as beginning after the failure — losing exactly the
+#: fact FR-003 exists to preserve. ``pending`` is provisional and likewise
+#: neither joins nor breaks streaks.
+_STREAK_BREAKING_STATUSES: frozenset[str] = frozenset(
+    {STATUS_CONFIRMED, STATUS_TARGET_ABSENT}
+)
+
+#: Rows deleted per statement, matching the purge task's batch convention.
+BATCH_SIZE: int = 500
+#: One shared budget for the whole run across all three categories; the
+#: remaining backlog carries over to the next scheduled run (FR-004/SC-004).
+MAX_BATCHES_PER_RUN: int = 10
+
+OPERATIONAL_RETENTION_KEY: str = "PURGE_AUDIT_OPERATIONAL_RETENTION_DAYS"
+EVIDENCE_RETENTION_KEY: str = "PURGE_AUDIT_EVIDENCE_RETENTION_DAYS"
+
+
+class ResolvedWindow(NamedTuple):
+    """A retention window resolved from config.
+
+    Distinguishes the three outcomes a caller must tell apart: a usable
+    window, the deliberate "off" default, and operator error. Collapsing
+    the last two into a bare ``None`` would force every caller to re-read
+    config to find out which it got.
+    """
+
+    days: int | None
+    invalid_key: str | None = None
+
+
+def _validated_window(key: str, value: Any) -> ResolvedWindow:
+    """Validate a configured day count, failing closed on anything odd.
+
+    An invalid value disables its category for the run rather than widening
+    removal (FR-005/SC-005).
+    """
+    # bool is an int subclass and floats would silently truncate — both are
+    # config mistakes, not day counts, on a knob that deletes rows.
+    if not isinstance(value, bool) and not isinstance(value, float):
+        try:
+            days: int = int(value)
+        except (TypeError, ValueError):
+            days = 0
+        if days > 0:
+            return ResolvedWindow(days)
+    logger.warning(
+        "prune_audit: invalid %s=%r; skipping this category for the run "
+        "(pruning never widens on bad configuration)",
+        key,
+        value,
+    )
+    return ResolvedWindow(None, key)
+
+
+def resolve_operational_retention_days() -> ResolvedWindow:
+    """The operational retention window, or a disabled window when invalid."""
+    return _validated_window(
+        OPERATIONAL_RETENTION_KEY, current_app.config.get(OPERATIONAL_RETENTION_KEY)
+    )
+
+
+def resolve_evidence_retention_days() -> ResolvedWindow:
+    """The evidence expiration window; disabled unless explicitly opted in.
+
+    Unset is the documented "never expire evidence" default (FR-006), not an
+    error, so it produces a disabled window with no warning.
+    """
+    value: Any = current_app.config.get(EVIDENCE_RETENTION_KEY)
+    if value is None:
+        return ResolvedWindow(None)
+    return _validated_window(EVIDENCE_RETENTION_KEY, value)
+
+
+@dataclass
+class PruneRunResult:
+    """Per-category removal counts and run disposition for one pruning run."""
+
+    blocked_duplicates: int = 0
+    operational_expired: int = 0
+    evidence_expired: int = 0
+    #: True when the shared batch budget ran out before every category's
+    #: candidates were drained; the remainder converges on later runs.
+    carried_over: bool = False
+    invalid_config_keys: list[str] = field(default_factory=list)
+
+    @property
+    def total_removed(self) -> int:
+        """Total rows removed across every category."""
+        return (
+            self.blocked_duplicates + self.operational_expired + self.evidence_expired
+        )
+
+    def as_dict(self) -> dict[str, Any]:
+        """The task-return / log-line shape of this result."""
+        return {
+            "removed": {
+                "blocked_duplicates": self.blocked_duplicates,
+                "operational_expired": self.operational_expired,
+                "evidence_expired": self.evidence_expired,
+            },
+            "carried_over": self.carried_over,
+            "invalid_config_keys": list(self.invalid_config_keys),
+        }
+
+
+def _streak_boundary_subquery(now: datetime) -> sa.Subquery:
+    """Per entity, the ``created_on`` of its newest streak-breaking row.
+
+    Entities absent from this subquery have never been proven destroyed —
+    all their blocked rows form one current streak. Future-dated rows are
+    excluded so a skewed writer clock cannot push the boundary ahead of a
+    live streak and make its rows look resolved.
+    """
+    table: sa.Table = PurgeAuditLog.__table__
+    return (
+        sa.select(
+            table.c.entity_type.label("entity_type"),
+            table.c.entity_uuid.label("entity_uuid"),
+            sa.func.max(table.c.created_on).label("boundary"),
+        )
+        .where(table.c.status.in_(_STREAK_BREAKING_STATUSES))
+        .where(table.c.entity_uuid.is_not(None))
+        .where(table.c.created_on <= now)
+        .group_by(table.c.entity_type, table.c.entity_uuid)
+        .subquery("streak_boundary")
+    )
+
+
+def _with_boundary(table: sa.Table, boundary: sa.Subquery) -> sa.Join:
+    """Outer-join each row to its entity's streak boundary (NULL if none)."""
+    return table.outerjoin(
+        boundary,
+        sa.and_(
+            table.c.entity_type == boundary.c.entity_type,
+            table.c.entity_uuid == boundary.c.entity_uuid,
+        ),
+    )
+
+
+def _in_current_streak(
+    row: sa.FromClause, boundary: sa.Subquery
+) -> sa.ColumnElement[bool]:
+    """Whether ``row`` is newer than its entity's boundary (or there is none).
+
+    Strictly newer: a row tied with the boundary sits on its resolved side.
+    A boundary proves the object was gone at that instant, and a block that
+    cannot be ordered after the destruction must not seed a new "current"
+    streak — that would mint a survivor exempt from age-out forever for an
+    object that no longer exists.
+    """
+    return sa.or_(boundary.c.boundary.is_(None), row.c.created_on > boundary.c.boundary)
+
+
+def _repeats_an_earlier_block(
+    table: sa.Table, boundary: sa.Subquery
+) -> sa.ColumnElement[bool]:
+    """Whether a same-reason current-streak block precedes this row with no
+    change of reason in between.
+
+    This is the audit writer's suppression rule
+    (:func:`audit.finalize_retention_blocked`) applied retroactively: a
+    block repeating the reason of the block just before it adds nothing,
+    while the first block after a reason change is the only durable record
+    of the new cause and is retained. The comparison is NULL-safe, so
+    consecutive reason-less (pre-feature) rows count as one run, deduped to
+    the run's earliest, and the first coded block ends that run. Note this
+    is *stricter* than the writer for reason-less rows: the writer never
+    suppresses a reason-less block (``_suppress_redundant_block`` bails on a
+    missing code), so here the pruner additionally collapses legacy
+    pre-feature duplicates — the streak's earliest "blocked since" row is
+    still always kept. Tied same-reason rows are not "earlier" than each
+    other, so all of them are kept.
+    """
+    earlier: sa.FromClause = table.alias("earlier_block")
+    between: sa.FromClause = table.alias("reason_change")
+    reason_changed_between: sa.ColumnElement[bool] = sa.exists(
+        sa.select(sa.literal(1))
+        .select_from(between)
+        .where(
+            sa.and_(
+                between.c.status == STATUS_BLOCKED,
+                between.c.entity_type == table.c.entity_type,
+                between.c.entity_uuid == table.c.entity_uuid,
+                # Inclusive bounds: a differing-reason block sharing an
+                # exact timestamp with either endpoint still breaks the run,
+                # so a reason-transition row tied with a neighbour is
+                # preserved as a run head rather than pruned as a repeat
+                # (the same preserving-side tie rule the pending and evidence
+                # guards use). Inclusive bounds only ever add boundaries —
+                # i.e. only ever preserve more, never delete more.
+                between.c.created_on >= earlier.c.created_on,
+                between.c.created_on <= table.c.created_on,
+                between.c.reason.is_distinct_from(table.c.reason),
+            )
+        )
+        .correlate(table, earlier)
+    )
+    return sa.exists(
+        sa.select(sa.literal(1))
+        .select_from(earlier)
+        .where(
+            sa.and_(
+                earlier.c.status == STATUS_BLOCKED,
+                earlier.c.entity_type == table.c.entity_type,
+                earlier.c.entity_uuid == table.c.entity_uuid,
+                _in_current_streak(earlier, boundary),
+                earlier.c.created_on < table.c.created_on,
+                earlier.c.reason.is_not_distinct_from(table.c.reason),
+                sa.not_(reason_changed_between),
+            )
+        )
+        .correlate(table, boundary)
+    )
+
+
+def _preceded_by_unresolved_attempt(table: sa.Table) -> sa.ColumnElement[bool]:
+    """Whether an unresolved (``pending``) attempt precedes this row.
+
+    A ``pending`` row is the only thing that can insert a streak boundary
+    into *history*: every other write lands at ``now``, newer than every
+    existing row, whereas reconciliation and the purge path finalize a
+    pending row **in place**, keeping its original ``created_on``. So a
+    pending row sitting between two blocked rows is a boundary that may
+    appear at any moment, and the blocked row after it would become the new
+    streak's survivor — the very row pruning must never delete.
+
+    A tied timestamp counts as preceding. Which of the two writes landed
+    first is unknowable from the row, the block's classification (current
+    duplicate, or resolved-streak row on an age window) changes with the
+    attempt's outcome, and deferring it until then costs nothing.
+    """
+    pending: sa.FromClause = table.alias("unresolved_attempt")
+    return sa.exists(
+        sa.select(sa.literal(1))
+        .select_from(pending)
+        .where(
+            sa.and_(
+                pending.c.status == STATUS_PENDING,
+                pending.c.entity_type == table.c.entity_type,
+                pending.c.entity_uuid == table.c.entity_uuid,
+                pending.c.created_on <= table.c.created_on,
+            )
+        )
+    )
+
+
+def _duplicate_candidates(now: datetime, limit: int) -> sa.sql.Select:
+    """Select current-streak blocked rows that repeat the block before them.
+
+    Age-independent by design (FR-003): a repeat is prunable the moment the
+    streak holds an earlier same-reason block, regardless of the retention
+    window. Trigger is not a discriminator — ``scheduled`` and ``force``
+    blocked rows share streaks. Reason *is*: the first block after a reason
+    change survives alongside the streak's earliest row (see
+    :func:`_repeats_an_earlier_block`).
+
+    Rows preceded by an unresolved attempt are skipped: their classification
+    is not stable, because that attempt can finalize into a boundary and
+    promote them to survivor between selection and deletion. Such rows are
+    collected once the attempt resolves. Note that pruning does not depend on
+    that happening promptly — reconciliation runs from the purge task, which
+    a deployment may have disabled or left in dry-run — so a long-lived
+    pending row defers its successors indefinitely rather than risking them.
+    """
+    table: sa.Table = PurgeAuditLog.__table__
+    boundary: sa.Subquery = _streak_boundary_subquery(now)
+    return (
+        sa.select(table.c.id)
+        .select_from(_with_boundary(table, boundary))
+        .where(table.c.status == STATUS_BLOCKED)

Review Comment:
   This can erase a force-purge audit record even though force attempts are documented as independent. A blocked force attempt with the same entity and reason after a scheduled block matches this selector because it does not restrict `trigger`; should force records be excluded from pruning?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org


Re: [PR] feat(deletion-retention): scheduled pruning policy for the purge audit log [superset]

Posted by "mikebridge (via GitHub)" <gi...@apache.org>.
mikebridge commented on PR #43490:
URL: https://github.com/apache/superset/pull/43490#issuecomment-5500242020

   Heads-up for re-review: I ran a focused data-systems/python/sqlalchemy review over the last delta (the reason-transition dedup + tie policy, `79ebceba18..86e86b4078`) and it surfaced one real edge case, now fixed in `a24a1313fe`:
   
   - **Differing-reason timestamp tie could delete a reason-transition run-head.** `_repeats_an_earlier_block` used strict `>`/`<` bounds, so a differing-reason block sharing an exact `created_on` with a run's earlier endpoint wasn't counted as a boundary — with `A(X)@t0`, a tied `A(X)@t1`/`B(Y)@t1`, and a later `A(X)@t2`, the last A (the first block after the reason returned to X) was pruned as a duplicate, dropping a reason-transition record. Fixed by making the boundary bounds inclusive (`>=`/`<=`), matching the preserving-side tie rule the pending and evidence guards already use; inclusive bounds can only ever preserve more, never delete more. New `test_a_tied_reason_change_still_breaks_the_run` fails on the old bounds and passes on the fix.
   
   Also in the same amend: corrected a docstring that overclaimed writer-parity for reason-less (pre-feature) runs (the pruner is stricter there — it collapses legacy duplicates; the writer never suppresses a reason-less block), and added an A,A,B,A,A reason-return test. Integration suite 28/28 on Postgres + SQLite, unit 83/83, pre-commit clean.
   
   Sorry to bounce the approval — flagging so it gets a fresh look at the new head.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org


Re: [PR] feat(deletion-retention): scheduled pruning policy for the purge audit log [superset]

Posted by "mikebridge (via GitHub)" <gi...@apache.org>.
mikebridge commented on PR #43490:
URL: https://github.com/apache/superset/pull/43490#issuecomment-5497673848

   Pushed 86e86b4078 addressing the three open threads (reason transitions now survive pruning, timestamp ties resolve on the preserving side, non-boolean `PURGE_AUDIT_PRUNING_ENABLED` warns at startup) — details on each thread. @sadpandajoe @rusackas ready for another look when you have a moment.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org


Re: [PR] feat(deletion-retention): scheduled pruning policy for the purge audit log [superset]

Posted by "mikebridge (via GitHub)" <gi...@apache.org>.
mikebridge commented on code in PR #43490:
URL: https://github.com/apache/superset/pull/43490#discussion_r3906790301


##########
superset/commands/deletion_retention/prune_audit.py:
##########
@@ -0,0 +1,607 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Deletion-only pruning of the ``purge_audit_log`` table.
+
+Bounds the audit history's growth without ever weakening its evidentiary
+value. Three delete categories, applied in priority order under one shared
+per-run batch budget:
+
+1. **Blocked duplicates** — within an entity's *current* blockage streak
+   (its ``blocked`` rows newer than the entity's newest streak-breaking
+   row), only the streak's earliest row survives: later duplicates are
+   removed regardless of age. The survivor carries the "blocked since"
+   fact and is never deleted while the streak is current.
+2. **Operational expiry** — ``blocked`` rows of *resolved* streaks and
+   ``failed`` rows older than ``PURGE_AUDIT_OPERATIONAL_RETENTION_DAYS``
+   age out.
+3. **Evidence expiry** — ``confirmed`` / ``target_absent`` rows are
+   untouchable unless ``PURGE_AUDIT_EVIDENCE_RETENTION_DAYS`` is
+   explicitly set (the operator's compliance assertion), and then only
+   rows older than that window.
+
+``pending`` rows belong to :func:`audit.reconcile_pending` and rows with
+future timestamps (clock skew) are excluded from every category *and* from
+streak classification, so a skewed writer cannot reclassify a live streak.
+
+Three invariants keep the survivor safe without a distributed lock, which
+matters because runs can overlap and both ``reconcile_pending`` and the
+purge path finalize rows concurrently. A duplicate is only ever deleted
+when no concurrent transition could turn it into a survivor first:
+
+* **A boundary is never removed while it still bounds anything.** Evidence
+  expiry refuses to delete a row while an older ``blocked`` or ``pending``
+  row for the same entity survives. Boundaries therefore never *recede*,
+  which would otherwise promote resolved rows into a current streak.
+* **A blocked row whose classification is unstable is never deleted.**
+  Finalizing a ``pending`` row resolves it *in place*, keeping its original
+  timestamp, so an unresolved attempt is a boundary that may appear
+  mid-history at any moment — and the blocked row after it would become its
+  streak's survivor. Blocked rows preceded by an unresolved attempt are
+  therefore skipped by **both** the duplicate and the operational category.
+  Age does not stabilize the classification: an old blocked row inside a
+  live streak is exactly the "blocked for years" case FR-009 protects.
+  Without this, a boundary moving *forward* would demote the current
+  survivor and promote the next row into its place — possibly a row already
+  selected for deletion.
+* **Deletes are conditional and counted from rowcounts.** A row whose
+  status changed since selection is not matched, so overlapping runs can
+  neither double-remove nor double-report.
+
+Boundaries moving forward past *all* of an entity's blocked rows leave no
+survivor to promote, so a selected duplicate may be removed slightly ahead
+of its retention window in that case. It was redundant either way and the
+streak's earliest row is untouched.
+
+Candidate selection is embedded in each ``DELETE`` statement. The derived-table
+wrapper keeps that shape legal on MySQL while ensuring that a pending or
+recovered row committed before the delete is evaluated participates in the
+survivor and boundary predicates. No stale list of candidate ids crosses a
+transaction boundary.
+
+Audit creation/recovery and every pruning batch take the same singleton
+database write lock before assigning a timestamp or evaluating candidates.
+The lock is held through commit, so an audit row cannot become visible in an
+already-processed logical past. Automatic pruning still ships disabled by
+default so operators explicitly choose their retention policy.
+"""
+
+from __future__ import annotations
+
+import logging
+from collections.abc import Callable
+from dataclasses import dataclass, field
+from datetime import datetime, timedelta
+from functools import partial
+from typing import Any, Literal, NamedTuple, TypeAlias
+
+import sqlalchemy as sa
+from flask import current_app
+
+from superset import db
+from superset.commands.deletion_retention.audit import (
+    acquire_coordination_lock,
+    utc_now,
+)
+from superset.models.purge_audit_log import (
+    PurgeAuditLog,
+    STATUS_BLOCKED,
+    STATUS_CONFIRMED,
+    STATUS_FAILED,
+    STATUS_PENDING,
+    STATUS_TARGET_ABSENT,
+)
+
+logger: logging.Logger = logging.getLogger(__name__)
+
+#: Operational records: noise-prone outcomes whose compliance value decays —
+#: a blocked or failed purge leaves the object in place (FR-001).
+OPERATIONAL_STATUSES: frozenset[str] = frozenset({STATUS_BLOCKED, STATUS_FAILED})
+#: Protected evidence: the only surviving trace of a destroyed object.
+PROTECTED_STATUSES: frozenset[str] = frozenset({STATUS_CONFIRMED, STATUS_TARGET_ABSENT})
+#: The outcomes that end a blockage streak — proof the object is gone.
+#:
+#: ``failed`` is deliberately absent. A failed purge is an infrastructure
+#: outcome (the cascade raised), not evidence the blockage cleared: the
+#: policy that blocked the entity is untouched, so the blockage continues
+#: across it. Treating ``failed`` as a boundary would let one transient
+#: error demote the "blocked since" survivor to an ageing duplicate and
+#: restate the blockage as beginning after the failure — losing exactly the
+#: fact FR-003 exists to preserve. ``pending`` is provisional and likewise
+#: neither joins nor breaks streaks.
+_STREAK_BREAKING_STATUSES: frozenset[str] = frozenset(
+    {STATUS_CONFIRMED, STATUS_TARGET_ABSENT}
+)
+
+#: Rows deleted per statement, matching the purge task's batch convention.
+BATCH_SIZE: int = 500
+#: One shared budget for the whole run across all three categories; the
+#: remaining backlog carries over to the next scheduled run (FR-004/SC-004).
+MAX_BATCHES_PER_RUN: int = 10
+
+OPERATIONAL_RETENTION_KEY: str = "PURGE_AUDIT_OPERATIONAL_RETENTION_DAYS"
+EVIDENCE_RETENTION_KEY: str = "PURGE_AUDIT_EVIDENCE_RETENTION_DAYS"
+
+
+class ResolvedWindow(NamedTuple):
+    """A retention window resolved from config.
+
+    Distinguishes the three outcomes a caller must tell apart: a usable
+    window, the deliberate "off" default, and operator error. Collapsing
+    the last two into a bare ``None`` would force every caller to re-read
+    config to find out which it got.
+    """
+
+    days: int | None
+    invalid_key: str | None = None
+
+
+def _validated_window(key: str, value: Any) -> ResolvedWindow:
+    """Validate a configured day count, failing closed on anything odd.
+
+    An invalid value disables its category for the run rather than widening
+    removal (FR-005/SC-005).
+    """
+    # bool is an int subclass and floats would silently truncate — both are
+    # config mistakes, not day counts, on a knob that deletes rows.
+    if not isinstance(value, bool) and not isinstance(value, float):
+        try:
+            days: int = int(value)
+        except (TypeError, ValueError):
+            days = 0
+        if days > 0:
+            return ResolvedWindow(days)
+    logger.warning(
+        "prune_audit: invalid %s=%r; skipping this category for the run "
+        "(pruning never widens on bad configuration)",
+        key,
+        value,
+    )
+    return ResolvedWindow(None, key)
+
+
+def resolve_operational_retention_days() -> ResolvedWindow:
+    """The operational retention window, or a disabled window when invalid."""
+    return _validated_window(
+        OPERATIONAL_RETENTION_KEY, current_app.config.get(OPERATIONAL_RETENTION_KEY)
+    )
+
+
+def resolve_evidence_retention_days() -> ResolvedWindow:
+    """The evidence expiration window; disabled unless explicitly opted in.
+
+    Unset is the documented "never expire evidence" default (FR-006), not an
+    error, so it produces a disabled window with no warning.
+    """
+    value: Any = current_app.config.get(EVIDENCE_RETENTION_KEY)
+    if value is None:
+        return ResolvedWindow(None)
+    return _validated_window(EVIDENCE_RETENTION_KEY, value)
+
+
+@dataclass
+class PruneRunResult:
+    """Per-category removal counts and run disposition for one pruning run."""
+
+    blocked_duplicates: int = 0
+    operational_expired: int = 0
+    evidence_expired: int = 0
+    #: True when the shared batch budget ran out before every category's
+    #: candidates were drained; the remainder converges on later runs.
+    carried_over: bool = False
+    invalid_config_keys: list[str] = field(default_factory=list)
+
+    @property
+    def total_removed(self) -> int:
+        """Total rows removed across every category."""
+        return (
+            self.blocked_duplicates + self.operational_expired + self.evidence_expired
+        )
+
+    def as_dict(self) -> dict[str, Any]:
+        """The task-return / log-line shape of this result."""
+        return {
+            "removed": {
+                "blocked_duplicates": self.blocked_duplicates,
+                "operational_expired": self.operational_expired,
+                "evidence_expired": self.evidence_expired,
+            },
+            "carried_over": self.carried_over,
+            "invalid_config_keys": list(self.invalid_config_keys),
+        }
+
+
+def _streak_boundary_subquery(now: datetime) -> sa.Subquery:
+    """Per entity, the ``created_on`` of its newest streak-breaking row.
+
+    Entities absent from this subquery have never been proven destroyed —
+    all their blocked rows form one current streak. Future-dated rows are
+    excluded so a skewed writer clock cannot push the boundary ahead of a
+    live streak and make its rows look resolved.
+    """
+    table: sa.Table = PurgeAuditLog.__table__
+    return (
+        sa.select(
+            table.c.entity_type.label("entity_type"),
+            table.c.entity_uuid.label("entity_uuid"),
+            sa.func.max(table.c.created_on).label("boundary"),
+        )
+        .where(table.c.status.in_(_STREAK_BREAKING_STATUSES))
+        .where(table.c.entity_uuid.is_not(None))
+        .where(table.c.created_on <= now)
+        .group_by(table.c.entity_type, table.c.entity_uuid)
+        .subquery("streak_boundary")
+    )
+
+
+def _survivor_subquery(now: datetime) -> sa.Subquery:
+    """Per entity, the ``created_on`` of its current streak's earliest row.
+
+    The survivor is the earliest ``blocked`` row strictly newer than the
+    entity's boundary (or its earliest blocked row outright when no boundary
+    exists). Rows sharing that exact timestamp are all treated as survivors —
+    with microsecond precision ties are pathological, and keeping an extra
+    row errs on the preserving side.
+    """
+    table: sa.Table = PurgeAuditLog.__table__
+    boundary: sa.Subquery = _streak_boundary_subquery(now)
+    return (
+        sa.select(
+            table.c.entity_type.label("entity_type"),
+            table.c.entity_uuid.label("entity_uuid"),
+            sa.func.min(table.c.created_on).label("survivor_created_on"),
+        )
+        .select_from(
+            table.outerjoin(
+                boundary,
+                sa.and_(
+                    table.c.entity_type == boundary.c.entity_type,
+                    table.c.entity_uuid == boundary.c.entity_uuid,
+                ),
+            )
+        )
+        .where(table.c.status == STATUS_BLOCKED)
+        .where(table.c.entity_uuid.is_not(None))
+        .where(table.c.created_on <= now)
+        .where(
+            sa.or_(
+                boundary.c.boundary.is_(None),
+                table.c.created_on > boundary.c.boundary,
+            )
+        )
+        .group_by(table.c.entity_type, table.c.entity_uuid)
+        .subquery("streak_survivor")
+    )
+
+
+def _preceded_by_unresolved_attempt(table: sa.Table) -> sa.ColumnElement[bool]:
+    """Whether an unresolved (``pending``) attempt precedes this row.
+
+    A ``pending`` row is the only thing that can insert a streak boundary
+    into *history*: every other write lands at ``now``, newer than every
+    existing row, whereas reconciliation and the purge path finalize a
+    pending row **in place**, keeping its original ``created_on``. So a
+    pending row sitting between two blocked rows is a boundary that may
+    appear at any moment, and the blocked row after it would become the new
+    streak's survivor — the very row pruning must never delete.
+    """
+    pending: sa.Table = table.alias("unresolved_attempt")
+    return sa.exists(
+        sa.select(sa.literal(1))
+        .select_from(pending)
+        .where(
+            sa.and_(
+                pending.c.status == STATUS_PENDING,
+                pending.c.entity_type == table.c.entity_type,
+                pending.c.entity_uuid == table.c.entity_uuid,
+                pending.c.created_on < table.c.created_on,
+            )
+        )
+    )
+
+
+def _duplicate_candidates(now: datetime, limit: int) -> sa.sql.Select:
+    """Select current-streak blocked rows that are not the streak survivor.
+
+    Age-independent by design (FR-003): a duplicate is prunable the moment a
+    streak has a survivor, regardless of the retention window. Trigger is not
+    a discriminator — ``scheduled`` and ``force`` blocked rows share streaks.
+
+    Rows preceded by an unresolved attempt are skipped: their classification
+    is not stable, because that attempt can finalize into a boundary and
+    promote them to survivor between selection and deletion. Such rows are
+    collected once the attempt resolves. Note that pruning does not depend on
+    that happening promptly — reconciliation runs from the purge task, which
+    a deployment may have disabled or left in dry-run — so a long-lived
+    pending row defers its successors indefinitely rather than risking them.
+    """
+    table: sa.Table = PurgeAuditLog.__table__
+    survivor: sa.Subquery = _survivor_subquery(now)
+    return (
+        sa.select(table.c.id)
+        .select_from(
+            table.join(
+                survivor,
+                sa.and_(
+                    table.c.entity_type == survivor.c.entity_type,
+                    table.c.entity_uuid == survivor.c.entity_uuid,
+                ),
+            )
+        )
+        .where(table.c.status == STATUS_BLOCKED)
+        .where(table.c.created_on > survivor.c.survivor_created_on)

Review Comment:
   Resolving now that the fix is pushed and green locally — please reopen if the run-head semantics aren't what you had in mind.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org


Re: [PR] feat(deletion-retention): scheduled pruning policy for the purge audit log [superset]

Posted by "bito-code-review[bot] (via GitHub)" <gi...@apache.org>.
bito-code-review[bot] commented on PR #43490:
URL: https://github.com/apache/superset/pull/43490#issuecomment-5499869813

   AI Code Review is in progress (usually takes 3 to 15 minutes unless it's a very large PR).
   
   <details>
   <summary>
   Bito Usage Guide
   </summary>
   
   **Commands**
         
   Type the following command in the pull request comment and save the comment.
         
   - `/review` - Manually triggers an incremental AI Review.
   - `/review full` - Manually triggers a full AI Review.
   
   - `/pause` - Pauses automatic reviews on this pull request.
   - `/resume` - Resumes automatic reviews.
   - `/resolve` - Marks all Bito-posted review comments as resolved.
   - `/abort` - Cancels all in-progress reviews.
   
         
   Refer to the <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/available-commands" target="_blank" rel="noopener noreferrer">documentation</a> for additional commands.
         
   **Configuration**
   
   This repository uses `Superset` You can customize the agent settings <a href="https://alpha.bito.ai/home/ai-agents/code-review-agent" target="_blank" rel="noopener noreferrer">here</a> or contact your Bito workspace admin at evan@preset.io.
         
   **Documentation & Help**
   - <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/getting-started/install-run-using-bito-cloud/create-or-customize-an-agent-instance" target="_blank" rel="noopener noreferrer">Customize agent settings</a>
   - <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/implementing-custom-code-review-rules" target="_blank" rel="noopener noreferrer">Review rules</a>
   - <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent" target="_blank" rel="noopener noreferrer">General documentation</a>
   - <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/faqs" target="_blank" rel="noopener noreferrer">FAQ</a>
   
   </details>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org


Re: [PR] feat(deletion-retention): scheduled pruning policy for the purge audit log [superset]

Posted by "bito-code-review[bot] (via GitHub)" <gi...@apache.org>.
bito-code-review[bot] commented on PR #43490:
URL: https://github.com/apache/superset/pull/43490#issuecomment-5544401811

   AI Code Review is in progress (usually takes 3 to 15 minutes unless it's a very large PR).
   
   <details>
   <summary>
   Bito Usage Guide
   </summary>
   
   **Commands**
         
   Type the following command in the pull request comment and save the comment.
         
   - `/review` - Manually triggers an incremental AI Review.
   - `/review full` - Manually triggers a full AI Review.
   
   - `/pause` - Pauses automatic reviews on this pull request.
   - `/resume` - Resumes automatic reviews.
   - `/resolve` - Marks all Bito-posted review comments as resolved.
   - `/abort` - Cancels all in-progress reviews.
   
         
   Refer to the <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/available-commands" target="_blank" rel="noopener noreferrer">documentation</a> for additional commands.
         
   **Configuration**
   
   This repository uses `Superset` You can customize the agent settings <a href="https://alpha.bito.ai/home/ai-agents/code-review-agent" target="_blank" rel="noopener noreferrer">here</a> or contact your Bito workspace admin at evan@preset.io.
         
   **Documentation & Help**
   - <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/getting-started/install-run-using-bito-cloud/create-or-customize-an-agent-instance" target="_blank" rel="noopener noreferrer">Customize agent settings</a>
   - <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/implementing-custom-code-review-rules" target="_blank" rel="noopener noreferrer">Review rules</a>
   - <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent" target="_blank" rel="noopener noreferrer">General documentation</a>
   - <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/faqs" target="_blank" rel="noopener noreferrer">FAQ</a>
   
   </details>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org


Re: [PR] feat(deletion-retention): scheduled pruning policy for the purge audit log [superset]

Posted by "mikebridge (via GitHub)" <gi...@apache.org>.
mikebridge commented on PR #43490:
URL: https://github.com/apache/superset/pull/43490#issuecomment-5401523173

   **Coordination note for whoever merges this:** #43485 (SC-115342, block-reason column) touches the same subsystem and four of the same files — `superset/commands/deletion_retention/audit.py`, `superset/models/purge_audit_log.py`, `superset/tasks/deletion_retention.py`, and `UPDATING.md`.
   
   The two changes are semantically independent: that one adds a nullable `reason` column with a migration; this one is deletion-only and adds no schema, so there is **no alembic chaining to manage** in either direction. The overlaps are textual only — here they are a public `utc_now` in `audit.py` (was `_utc_now`, now shared with the pruner so cutoffs and `created_on` cannot drift apart), an `ALL_STATUSES` constant on the model, a new task in `tasks/deletion_retention.py`, and a new `UPDATING.md` entry.
   
   Whichever lands second rebases; no ordering preference from this side. One thing worth knowing if the reason column ever grows the status vocabulary: this PR adds an exhaustiveness test asserting every status in `ALL_STATUSES` falls into exactly one retention category, so a new status without a category fails a test rather than silently never being pruned.
   
   *Comment by Claude (AI) on behalf of @mikebridge.*


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org


Re: [PR] feat(deletion-retention): scheduled pruning policy for the purge audit log [superset]

Posted by "bito-code-review[bot] (via GitHub)" <gi...@apache.org>.
bito-code-review[bot] commented on PR #43490:
URL: https://github.com/apache/superset/pull/43490#issuecomment-5401506754

   AI Code Review is in progress (usually takes 3 to 15 minutes unless it's a very large PR).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org


Re: [PR] feat(deletion-retention): scheduled pruning policy for the purge audit log [superset]

Posted by "mikebridge (via GitHub)" <gi...@apache.org>.
mikebridge commented on code in PR #43490:
URL: https://github.com/apache/superset/pull/43490#discussion_r3855201209


##########
superset/migrations/versions/2026-08-24_15-50_a6c21e5b4d93_index_purge_audit_pruning.py:
##########
@@ -0,0 +1,46 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Index purge-audit pruning predicates.
+
+Revision ID: a6c21e5b4d93
+Revises: 1072de5ed955
+Create Date: 2026-08-24 15:50:00.000000
+
+"""
+
+from superset.migrations.shared.utils import create_index, drop_index
+
+# revision identifiers, used by Alembic.
+revision: str = "a6c21e5b4d93"
+down_revision: str = "1072de5ed955"
+
+_TABLE_NAME: str = "purge_audit_log"
+_INDEX_NAME: str = "ix_purge_audit_log_pruning"
+
+
+def upgrade() -> None:
+    """Add an index matching recurring pruning access patterns."""
+    create_index(
+        _TABLE_NAME,
+        _INDEX_NAME,
+        ["status", "entity_type", "entity_uuid", "created_on"],
+    )
+
+
+def downgrade() -> None:
+    """Remove the purge-audit pruning index."""
+    drop_index(_TABLE_NAME, _INDEX_NAME)

Review Comment:
   Thanks — this migration is covered in .  verifies the table, index name, and ordered columns, and  verifies downgrade symmetry. The file also covers the immediately-following coordination migration; its filename emphasizes that migration, which may have made the index coverage easy to miss.



##########
tests/unit_tests/commands/deletion_retention/test_prune_audit.py:
##########
@@ -0,0 +1,294 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Unit tests for the pure parts of purge-audit pruning.
+
+Query behavior against real rows (streak survivors, batching, protection
+invariants) is covered by
+``tests/integration_tests/deletion_retention/prune_audit_tests.py``.
+"""
+
+from __future__ import annotations
+
+from contextlib import contextmanager
+from datetime import datetime
+from typing import Any, Iterator
+from unittest.mock import patch
+
+import pytest
+import sqlalchemy as sa
+from flask import current_app
+from sqlalchemy.dialects import mysql
+
+from superset.commands.deletion_retention import prune_audit
+from superset.commands.deletion_retention.prune_audit import (
+    EVIDENCE_RETENTION_KEY,
+    OPERATIONAL_RETENTION_KEY,
+    OPERATIONAL_STATUSES,
+    PROTECTED_STATUSES,
+    PruneRunResult,
+    resolve_evidence_retention_days,
+    resolve_operational_retention_days,
+)
+from superset.models.purge_audit_log import (
+    ALL_STATUSES,
+    STATUS_BLOCKED,
+    STATUS_CONFIRMED,
+    STATUS_FAILED,
+    STATUS_PENDING,
+    STATUS_TARGET_ABSENT,
+)
+
+_METRIC_PREFIX: str = "deletion_retention.prune_purge_audit"
+
+
+@contextmanager
+def without_config(key: str) -> Iterator[None]:
+    """Remove a config key for the duration of the block, then restore it.
+
+    ``patch.dict`` cannot express key *removal*, and the unit-test app
+    fixture is module-scoped, so a bare ``pop`` would leak to later tests.
+    """
+    with patch.dict(current_app.config):
+        current_app.config.pop(key, None)
+        yield
+
+
+def test_retention_categories_partition_every_status() -> None:
+    """Require every status to belong to a retention category.
+
+    A new status added to the model without a category would silently never
+    be pruned; this assertion catches that omission.
+    """
+    assert OPERATIONAL_STATUSES == {STATUS_BLOCKED, STATUS_FAILED}
+    assert PROTECTED_STATUSES == {STATUS_CONFIRMED, STATUS_TARGET_ABSENT}
+    assert not OPERATIONAL_STATUSES & PROTECTED_STATUSES
+    assert OPERATIONAL_STATUSES | PROTECTED_STATUSES | {STATUS_PENDING} == ALL_STATUSES
+
+
+def test_only_proof_of_destruction_breaks_a_blockage_streak() -> None:
+    """Require destruction evidence to break a blockage streak.
+
+    A failed attempt is an infrastructure outcome, and pending is provisional.
+    Neither proves that the blockage cleared.
+    """
+    assert prune_audit._STREAK_BREAKING_STATUSES == {
+        STATUS_CONFIRMED,
+        STATUS_TARGET_ABSENT,
+    }
+    assert STATUS_FAILED not in prune_audit._STREAK_BREAKING_STATUSES
+    assert STATUS_PENDING not in prune_audit._STREAK_BREAKING_STATUSES
+
+
+def test_operational_retention_defaults_to_ninety_days() -> None:
+    """Use the shipped 90-day operational retention default."""
+    assert current_app.config[OPERATIONAL_RETENTION_KEY] == 90
+    assert resolve_operational_retention_days().days == 90
+
+
+@pytest.mark.parametrize("value", [30, 1, 36500])
+def test_operational_retention_accepts_positive_days(value: int) -> None:
+    with patch.dict(current_app.config, {OPERATIONAL_RETENTION_KEY: value}):
+        assert resolve_operational_retention_days() == (value, None)
+
+
+@pytest.mark.parametrize("value", [0, -5, True, False, "ninety", None, 1.5])
+def test_operational_retention_fails_closed_on_invalid_values(value: Any) -> None:
+    """Disable a category and identify its invalid configuration key."""
+    with patch.dict(current_app.config, {OPERATIONAL_RETENTION_KEY: value}):
+        window: prune_audit.ResolvedWindow = resolve_operational_retention_days()
+    assert window.days is None
+    assert window.invalid_key == OPERATIONAL_RETENTION_KEY
+
+
+def test_missing_operational_key_is_reported_as_invalid_not_assumed() -> None:
+    """A popped key is operator error, not a silent 90-day assumption."""
+    with without_config(OPERATIONAL_RETENTION_KEY):
+        window: prune_audit.ResolvedWindow = resolve_operational_retention_days()
+    assert window.days is None
+    assert window.invalid_key == OPERATIONAL_RETENTION_KEY
+
+
+def test_evidence_retention_defaults_to_off_without_warning() -> None:
+    """Unset is the documented never-expire default (FR-006), not an error:
+    disabled, no warning, and no key reported as invalid."""
+    with without_config(EVIDENCE_RETENTION_KEY):
+        with patch.object(prune_audit, "logger") as mock_logger:
+            window: prune_audit.ResolvedWindow = resolve_evidence_retention_days()
+        mock_logger.warning.assert_not_called()
+    assert window == (None, None)
+
+
+def test_evidence_retention_accepts_the_explicit_opt_in() -> None:
+    with patch.dict(current_app.config, {EVIDENCE_RETENTION_KEY: 3650}):
+        window: prune_audit.ResolvedWindow = resolve_evidence_retention_days()
+    assert window.days == 3650
+
+
+@pytest.mark.parametrize("value", [0, -1, True, "forever"])
+def test_evidence_retention_fails_closed_on_invalid_values(value: Any) -> None:
+    with patch.dict(current_app.config, {EVIDENCE_RETENTION_KEY: value}):
+        with patch.object(prune_audit, "logger") as mock_logger:
+            window: prune_audit.ResolvedWindow = resolve_evidence_retention_days()
+        mock_logger.warning.assert_called_once()
+    assert window.days is None
+    assert window.invalid_key == EVIDENCE_RETENTION_KEY
+
+
+def test_prune_run_result_totals_and_dict_shape() -> None:
+    result: PruneRunResult = PruneRunResult(
+        blocked_duplicates=3, operational_expired=2, evidence_expired=1
+    )
+    assert result.total_removed == 6
+    assert result.as_dict() == {
+        "removed": {
+            "blocked_duplicates": 3,
+            "operational_expired": 2,
+            "evidence_expired": 1,
+        },
+        "carried_over": False,
+        "invalid_config_keys": [],
+    }
+
+
+def test_disabled_task_reports_itself_and_removes_nothing() -> None:
+    """Report a disabled run without reaching the prune implementation."""
+    from superset.tasks import deletion_retention as task_module
+
+    with patch.dict(current_app.config, {"PURGE_AUDIT_PRUNING_ENABLED": False}):
+        with (
+            patch.object(task_module, "stats_logger_manager") as mock_stats,
+            patch.object(task_module, "logger") as mock_logger,
+            patch.object(task_module.prune_audit, "run_prune") as mock_run,
+        ):
+            outcome: dict[str, Any] = task_module.prune_purge_audit()
+    assert outcome == {"skipped_disabled": 1}
+    mock_run.assert_not_called()
+    mock_stats.instance.incr.assert_called_once_with(
+        f"{_METRIC_PREFIX}.skipped_disabled"
+    )
+    assert mock_logger.info.called
+
+
+@pytest.mark.parametrize("value", ["false", "0", 1, None])
+def test_non_boolean_master_switch_fails_closed(value: Any) -> None:
+    """Never interpret truthy strings or numeric values as deletion opt-in."""
+    from superset.tasks import deletion_retention as task_module
+
+    with patch.dict(current_app.config, {"PURGE_AUDIT_PRUNING_ENABLED": value}):
+        with (
+            patch.object(task_module, "stats_logger_manager") as mock_stats,
+            patch.object(task_module.prune_audit, "run_prune") as mock_run,
+        ):
+            outcome: dict[str, Any] = task_module.prune_purge_audit()
+
+    assert outcome == {"skipped_invalid_config": 1}
+    mock_run.assert_not_called()
+    mock_stats.instance.incr.assert_called_once_with(
+        f"{_METRIC_PREFIX}.skipped_invalid_config"
+    )
+
+
+def test_failed_run_is_isolated_rolled_back_and_distinguishable() -> None:
+    """Report and isolate a failed pruning run.
+
+    The task rolls back and returns an error marker that cannot be mistaken
+    for a successful run that removed nothing.
+    """
+    from superset.tasks import deletion_retention as task_module
+
+    with patch.dict(current_app.config, {"PURGE_AUDIT_PRUNING_ENABLED": True}):
+        with (
+            patch.object(task_module, "stats_logger_manager") as mock_stats,
+            patch.object(task_module, "db") as mock_db,
+            patch.object(
+                task_module.prune_audit,
+                "run_prune",
+                side_effect=RuntimeError("boom"),
+            ),
+        ):
+            outcome: dict[str, Any] = task_module.prune_purge_audit()
+    assert outcome == {"error": 1}
+    mock_db.session.rollback.assert_called_once()
+    mock_stats.instance.incr.assert_called_once_with(f"{_METRIC_PREFIX}.failed")
+
+
+def test_successful_run_mirrors_counts_and_carryover_into_metrics() -> None:
+    """Expose category counts and convergence through metrics."""
+    from superset.tasks import deletion_retention as task_module
+
+    fake: PruneRunResult = PruneRunResult(
+        blocked_duplicates=7,
+        operational_expired=4,
+        evidence_expired=0,
+        carried_over=True,
+    )

Review Comment:
   Thanks. I’m leaving these two small setups local: they exercise distinct task outcomes and their patch contexts make each test’s boundary explicit. Extracting a shared fixture here would add indirection without removing meaningful maintenance burden, and as noted this is not duplication introduced by the reviewed change.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org


Re: [PR] feat(deletion-retention): scheduled pruning policy for the purge audit log [superset]

Posted by "codeant-ai-for-open-source[bot] (via GitHub)" <gi...@apache.org>.
codeant-ai-for-open-source[bot] commented on code in PR #43490:
URL: https://github.com/apache/superset/pull/43490#discussion_r3857012751


##########
superset/initialization/__init__.py:
##########
@@ -1002,11 +1005,14 @@ def _warn_if_retention_beat_missing(self) -> None:
                 "CeleryConfig or add the module to your override.",
                 self._RETENTION_TASK_MODULE,
             )
-        default_flags = self.config.get("DEFAULT_FEATURE_FLAGS", {})
-        configured_flags = self.config.get("FEATURE_FLAGS", {})
-        soft_delete_enabled = bool(
+        default_flags: dict[str, Any] = self.config.get("DEFAULT_FEATURE_FLAGS", {})
+        configured_flags: dict[str, Any] = self.config.get("FEATURE_FLAGS", {})
+        soft_delete_enabled: bool = bool(
             configured_flags.get("SOFT_DELETE", default_flags.get("SOFT_DELETE", False))
         )
+        audit_pruning_enabled: bool = (
+            self.config.get("PURGE_AUDIT_PRUNING_ENABLED", False) is True
+        )

Review Comment:
   **Suggestion:** Invalid values such as the string `"true"` or integer `1` are treated as disabled here because the diagnostic requires identity with the singleton `True`. The task itself rejects these values as invalid and skips pruning, but startup emits no warning about the invalid master switch or the missing schedule, so an operator typo can leave audit pruning permanently inactive without an actionable initialization warning. Validate and report invalid non-boolean values separately before deciding whether to check the schedule. [incorrect condition logic]
   
   <details>
   <summary><b>Severity Level:</b> Minor 🧹</summary>
   
   ```mdx
   - ⚠️ Startup diagnostics omit invalid pruning configuration.
   - ⚠️ Missing audit-prune scheduling can remain unnoticed until beat execution.
   - ⚠️ Purge audit rows continue accumulating when pruning is inactive.
   ```
   </details>
   
   [![Use CodeAnt Skill](https://new-codeant-butcket.s3.us-west-1.amazonaws.com/badges/use-codeant-skill-flat-v2.svg)](https://docs.codeant.ai/cli/resolve-pr-comments-skill) [![Fix in Cursor](https://new-codeant-butcket.s3.us-west-1.amazonaws.com/badges/fix-in-cursor-flat.svg)](https://app.codeant.ai/fix-in-ide?tool=cursor&prompt_id=654c34c392a047aea55d99f4688cd2df&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset) [![Fix in VSCode Claude](https://new-codeant-butcket.s3.us-west-1.amazonaws.com/badges/fix-in-vscode-claude-flat.svg)](https://app.codeant.ai/fix-in-ide?tool=vscode-claude&prompt_id=654c34c392a047aea55d99f4688cd2df&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset)
   <details>
   <summary><b>Prompt for AI Agent 🤖 </b></summary>
   
   ```mdx
   This is a comment left during a code review.
   
   **Path:** superset/initialization/__init__.py
   **Line:** 1013:1015
   **Comment:**
   	*Incorrect Condition Logic: Invalid values such as the string `"true"` or integer `1` are treated as disabled here because the diagnostic requires identity with the singleton `True`. The task itself rejects these values as invalid and skips pruning, but startup emits no warning about the invalid master switch or the missing schedule, so an operator typo can leave audit pruning permanently inactive without an actionable initialization warning. Validate and report invalid non-boolean values separately before deciding whether to check the schedule.
   
   Validate the correctness of the flagged issue. If correct, How can I resolve this? If you propose a fix, implement it and please make it concise.
   Once fix is implemented, also check other comments on the same PR, and ask user if the user wants to fix the rest of the comments as well. if said yes, then fetch all the comments validate the correctness and implement a minimal fix
   ```
   </details>
   <a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F43490&comment_hash=cd7bc418d4512ac3a999686b4cbefbae0158f1f056db743c2c63dfa152adde75&reaction=like'>👍</a> | <a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F43490&comment_hash=cd7bc418d4512ac3a999686b4cbefbae0158f1f056db743c2c63dfa152adde75&reaction=dislike'>👎</a>



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org


Re: [PR] feat(deletion-retention): scheduled pruning policy for the purge audit log [superset]

Posted by "codecov[bot] (via GitHub)" <gi...@apache.org>.
codecov[bot] commented on PR #43490:
URL: https://github.com/apache/superset/pull/43490#issuecomment-5401691103

   ## [Codecov](https://app.codecov.io/gh/apache/superset/pull/43490?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
   :x: Patch coverage is `84.07643%` with `25 lines` in your changes missing coverage. Please review.
   :white_check_mark: Project coverage is 66.08%. Comparing base ([`1bde62f`](https://app.codecov.io/gh/apache/superset/commit/1bde62f997378987d19313a003a625502213fa19?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`e2e624f`](https://app.codecov.io/gh/apache/superset/commit/e2e624ffac7876027eb3a9d4b6ba342a08ed6031?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
   :warning: Report is 16 commits behind head on master.
   
   | [Files with missing lines](https://app.codecov.io/gh/apache/superset/pull/43490?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
   |---|---|---|
   | [superset/tasks/deletion\_retention.py](https://app.codecov.io/gh/apache/superset/pull/43490?src=pr&el=tree&filepath=superset%2Ftasks%2Fdeletion_retention.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-c3VwZXJzZXQvdGFza3MvZGVsZXRpb25fcmV0ZW50aW9uLnB5) | 19.04% | [17 Missing :warning: ](https://app.codecov.io/gh/apache/superset/pull/43490?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
   | [...uperset/commands/deletion\_retention/prune\_audit.py](https://app.codecov.io/gh/apache/superset/pull/43490?src=pr&el=tree&filepath=superset%2Fcommands%2Fdeletion_retention%2Fprune_audit.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-c3VwZXJzZXQvY29tbWFuZHMvZGVsZXRpb25fcmV0ZW50aW9uL3BydW5lX2F1ZGl0LnB5) | 94.44% | [5 Missing and 2 partials :warning: ](https://app.codecov.io/gh/apache/superset/pull/43490?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
   | [superset/initialization/\_\_init\_\_.py](https://app.codecov.io/gh/apache/superset/pull/43490?src=pr&el=tree&filepath=superset%2Finitialization%2F__init__.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-c3VwZXJzZXQvaW5pdGlhbGl6YXRpb24vX19pbml0X18ucHk=) | 66.66% | [0 Missing and 1 partial :warning: ](https://app.codecov.io/gh/apache/superset/pull/43490?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
   > :exclamation:  There is a different number of reports uploaded between BASE (1bde62f) and HEAD (e2e624f). Click for more details.
   > 
   > <details><summary>HEAD has 5 uploads less than BASE</summary>
   >
   >| Flag | BASE (1bde62f) | HEAD (e2e624f) |
   >|------|------|------|
   >|python|8|3|
   ></details>
   
   <details><summary>Additional details and impacted files</summary>
   
   
   
   ```diff
   @@             Coverage Diff             @@
   ##           master   #43490       +/-   ##
   ===========================================
   - Coverage   78.90%   66.08%   -12.83%     
   ===========================================
     Files        2878     2879        +1     
     Lines      165031   165309      +278     
     Branches    38135    38168       +33     
   ===========================================
   - Hits       130218   109242    -20976     
   - Misses      32363    53843    +21480     
   + Partials     2450     2224      -226     
   ```
   
   | [Flag](https://app.codecov.io/gh/apache/superset/pull/43490/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
   |---|---|---|
   | [hive](https://app.codecov.io/gh/apache/superset/pull/43490/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `38.02% <40.76%> (-0.03%)` | :arrow_down: |
   | [mysql](https://app.codecov.io/gh/apache/superset/pull/43490/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
   | [postgres](https://app.codecov.io/gh/apache/superset/pull/43490/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
   | [presto](https://app.codecov.io/gh/apache/superset/pull/43490/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `39.94% <40.76%> (-0.04%)` | :arrow_down: |
   | [python](https://app.codecov.io/gh/apache/superset/pull/43490/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `57.67% <84.07%> (-25.90%)` | :arrow_down: |
   | [sqlite](https://app.codecov.io/gh/apache/superset/pull/43490/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `57.53% <84.07%> (+<0.01%)` | :arrow_up: |
   | [unit](https://app.codecov.io/gh/apache/superset/pull/43490/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
   
   Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
   </details>
   
   [:umbrella: View full report in Codecov by Harness](https://app.codecov.io/gh/apache/superset/pull/43490?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).   
   :loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
   <details><summary> :rocket: New features to boost your workflow: </summary>
   
   - :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
   - :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org


Re: [PR] feat(deletion-retention): scheduled pruning policy for the purge audit log [superset]

Posted by "sadpandajoe (via GitHub)" <gi...@apache.org>.
sadpandajoe commented on code in PR #43490:
URL: https://github.com/apache/superset/pull/43490#discussion_r3992450606


##########
superset/commands/deletion_retention/prune_audit.py:
##########
@@ -0,0 +1,726 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Deletion-only pruning of the ``purge_audit_log`` table.
+
+Bounds the audit history's growth without ever weakening its evidentiary
+value. Three delete categories, applied in priority order under one shared
+per-run batch budget:
+
+1. **Blocked duplicates** — within an entity's *current* blockage streak
+   (its ``blocked`` rows newer than the entity's newest streak-breaking
+   row), only the first row of each run of consecutive same-reason rows
+   survives: the streak's earliest row and the first row after every
+   change of block reason. Later same-reason repeats are removed
+   regardless of age. The survivors carry the "blocked since" fact and
+   the reason history — for coded reasons, the same rows the audit writer's
+   own suppression rule retains (reason-less legacy runs are additionally
+   collapsed to their earliest here) — and are never deleted while the
+   streak is current.
+2. **Operational expiry** — ``blocked`` rows of *resolved* streaks and
+   ``failed`` rows older than ``PURGE_AUDIT_OPERATIONAL_RETENTION_DAYS``
+   age out.
+3. **Evidence expiry** — ``confirmed`` / ``target_absent`` rows are
+   untouchable unless ``PURGE_AUDIT_EVIDENCE_RETENTION_DAYS`` is
+   explicitly set (the operator's compliance assertion), and then only
+   rows older than that window.
+
+``pending`` rows belong to :func:`audit.reconcile_pending` and rows with
+future timestamps (clock skew) are excluded from every category *and* from
+streak classification, so a skewed writer cannot reclassify a live streak.
+
+Three invariants keep the survivor safe without a distributed lock, which
+matters because runs can overlap and both ``reconcile_pending`` and the
+purge path finalize rows concurrently. A duplicate is only ever deleted
+when no concurrent transition could turn it into a survivor first:
+
+* **A boundary is never removed while it still bounds anything.** Evidence
+  expiry refuses to delete a row while an older ``blocked`` or ``pending``
+  row for the same entity survives. Boundaries therefore never *recede*,
+  which would otherwise promote resolved rows into a current streak.
+* **A blocked row whose classification is unstable is never deleted.**
+  Finalizing a ``pending`` row resolves it *in place*, keeping its original
+  timestamp, so an unresolved attempt is a boundary that may appear
+  mid-history at any moment — and the blocked row after it would become its
+  streak's survivor. Blocked rows preceded by an unresolved attempt are
+  therefore skipped by **both** the duplicate and the operational category.
+  Age does not stabilize the classification: an old blocked row inside a
+  live streak is exactly the "blocked for years" case FR-009 protects.
+  Without this, a boundary moving *forward* would demote the current
+  survivor and promote the next row into its place — possibly a row already
+  selected for deletion.
+* **Deletes are conditional and counted from rowcounts.** A row whose
+  status changed since selection is not matched, so overlapping runs can
+  neither double-remove nor double-report.
+
+Boundaries moving forward past *all* of an entity's blocked rows leave no
+survivor to promote, so a selected duplicate may be removed slightly ahead
+of its retention window in that case. It was redundant either way and the
+streak's earliest row is untouched.
+
+Timestamps order rows. Ties are possible — legacy second-precision rows,
+two writers within one clock tick — and are resolved on the preserving
+side: a ``pending`` row tied with a blocked row counts as preceding it (the
+block is deferred until the attempt resolves); a blocked row tied with a
+boundary sits on the boundary's *resolved* side (it ages out instead of
+seeding a new current streak, and the boundary is not removed before it);
+and tied same-reason blocked rows are all retained.
+
+Candidate selection is embedded in each ``DELETE`` statement. The derived-table
+wrapper keeps that shape legal on MySQL while ensuring that a pending or
+recovered row committed before the delete is evaluated participates in the
+survivor and boundary predicates. No stale list of candidate ids crosses a
+transaction boundary.
+
+Audit creation/recovery and every pruning batch take the same singleton
+database write lock before assigning a timestamp or evaluating candidates.
+The lock is held through commit, so an audit row cannot become visible in an
+already-processed logical past. Automatic pruning still ships disabled by
+default so operators explicitly choose their retention policy.
+"""
+
+from __future__ import annotations
+
+import logging
+from collections.abc import Callable
+from dataclasses import dataclass, field
+from datetime import datetime, timedelta
+from functools import partial
+from typing import Any, Literal, NamedTuple, TypeAlias
+
+import sqlalchemy as sa
+from flask import current_app
+
+from superset import db
+from superset.commands.deletion_retention.audit import (
+    acquire_coordination_lock,
+    TRIGGER_FORCE,
+    utc_now,
+)
+from superset.models.purge_audit_log import (
+    PurgeAuditLog,
+    STATUS_BLOCKED,
+    STATUS_CONFIRMED,
+    STATUS_FAILED,
+    STATUS_PENDING,
+    STATUS_TARGET_ABSENT,
+)
+
+logger: logging.Logger = logging.getLogger(__name__)
+
+#: Operational records: noise-prone outcomes whose compliance value decays —
+#: a blocked or failed purge leaves the object in place (FR-001).
+OPERATIONAL_STATUSES: frozenset[str] = frozenset({STATUS_BLOCKED, STATUS_FAILED})
+#: Protected evidence: the only surviving trace of a destroyed object.
+PROTECTED_STATUSES: frozenset[str] = frozenset({STATUS_CONFIRMED, STATUS_TARGET_ABSENT})
+#: The outcomes that end a blockage streak — proof the object is gone.
+#:
+#: ``failed`` is deliberately absent. A failed purge is an infrastructure
+#: outcome (the cascade raised), not evidence the blockage cleared: the
+#: policy that blocked the entity is untouched, so the blockage continues
+#: across it. Treating ``failed`` as a boundary would let one transient
+#: error demote the "blocked since" survivor to an ageing duplicate and
+#: restate the blockage as beginning after the failure — losing exactly the
+#: fact FR-003 exists to preserve. ``pending`` is provisional and likewise
+#: neither joins nor breaks streaks.
+_STREAK_BREAKING_STATUSES: frozenset[str] = frozenset(
+    {STATUS_CONFIRMED, STATUS_TARGET_ABSENT}
+)
+
+#: Rows deleted per statement, matching the purge task's batch convention.
+BATCH_SIZE: int = 500
+#: One shared budget for the whole run across all three categories; the
+#: remaining backlog carries over to the next scheduled run (FR-004/SC-004).
+MAX_BATCHES_PER_RUN: int = 10
+
+OPERATIONAL_RETENTION_KEY: str = "PURGE_AUDIT_OPERATIONAL_RETENTION_DAYS"
+EVIDENCE_RETENTION_KEY: str = "PURGE_AUDIT_EVIDENCE_RETENTION_DAYS"
+
+
+class ResolvedWindow(NamedTuple):
+    """A retention window resolved from config.
+
+    Distinguishes the three outcomes a caller must tell apart: a usable
+    window, the deliberate "off" default, and operator error. Collapsing
+    the last two into a bare ``None`` would force every caller to re-read
+    config to find out which it got.
+    """
+
+    days: int | None
+    invalid_key: str | None = None
+
+
+def _validated_window(key: str, value: Any) -> ResolvedWindow:
+    """Validate a configured day count, failing closed on anything odd.
+
+    An invalid value disables its category for the run rather than widening
+    removal (FR-005/SC-005).
+    """
+    # bool is an int subclass and floats would silently truncate — both are
+    # config mistakes, not day counts, on a knob that deletes rows.
+    if not isinstance(value, bool) and not isinstance(value, float):
+        try:
+            days: int = int(value)
+        except (TypeError, ValueError):
+            days = 0
+        if days > 0:
+            return ResolvedWindow(days)
+    logger.warning(
+        "prune_audit: invalid %s=%r; skipping this category for the run "
+        "(pruning never widens on bad configuration)",
+        key,
+        value,
+    )
+    return ResolvedWindow(None, key)
+
+
+def resolve_operational_retention_days() -> ResolvedWindow:
+    """The operational retention window, or a disabled window when invalid."""
+    return _validated_window(
+        OPERATIONAL_RETENTION_KEY, current_app.config.get(OPERATIONAL_RETENTION_KEY)
+    )
+
+
+def resolve_evidence_retention_days() -> ResolvedWindow:
+    """The evidence expiration window; disabled unless explicitly opted in.
+
+    Unset is the documented "never expire evidence" default (FR-006), not an
+    error, so it produces a disabled window with no warning.
+    """
+    value: Any = current_app.config.get(EVIDENCE_RETENTION_KEY)
+    if value is None:
+        return ResolvedWindow(None)
+    return _validated_window(EVIDENCE_RETENTION_KEY, value)
+
+
+@dataclass
+class PruneRunResult:
+    """Per-category removal counts and run disposition for one pruning run."""
+
+    blocked_duplicates: int = 0
+    operational_expired: int = 0
+    evidence_expired: int = 0
+    #: True when the shared batch budget ran out before every category's
+    #: candidates were drained; the remainder converges on later runs.
+    carried_over: bool = False
+    invalid_config_keys: list[str] = field(default_factory=list)
+
+    @property
+    def total_removed(self) -> int:
+        """Total rows removed across every category."""
+        return (
+            self.blocked_duplicates + self.operational_expired + self.evidence_expired
+        )
+
+    def as_dict(self) -> dict[str, Any]:
+        """The task-return / log-line shape of this result."""
+        return {
+            "removed": {
+                "blocked_duplicates": self.blocked_duplicates,
+                "operational_expired": self.operational_expired,
+                "evidence_expired": self.evidence_expired,
+            },
+            "carried_over": self.carried_over,
+            "invalid_config_keys": list(self.invalid_config_keys),
+        }
+
+
+def _streak_boundary_subquery(now: datetime) -> sa.Subquery:
+    """Per entity, the ``created_on`` of its newest streak-breaking row.
+
+    Entities absent from this subquery have never been proven destroyed —
+    all their blocked rows form one current streak. Future-dated rows are
+    excluded so a skewed writer clock cannot push the boundary ahead of a
+    live streak and make its rows look resolved.
+    """
+    table: sa.Table = PurgeAuditLog.__table__
+    return (
+        sa.select(
+            table.c.entity_type.label("entity_type"),
+            table.c.entity_uuid.label("entity_uuid"),
+            sa.func.max(table.c.created_on).label("boundary"),
+        )
+        .where(table.c.status.in_(_STREAK_BREAKING_STATUSES))
+        .where(table.c.entity_uuid.is_not(None))
+        .where(table.c.created_on <= now)
+        .group_by(table.c.entity_type, table.c.entity_uuid)
+        .subquery("streak_boundary")
+    )
+
+
+def _with_boundary(table: sa.Table, boundary: sa.Subquery) -> sa.Join:
+    """Outer-join each row to its entity's streak boundary (NULL if none)."""
+    return table.outerjoin(
+        boundary,
+        sa.and_(
+            table.c.entity_type == boundary.c.entity_type,
+            table.c.entity_uuid == boundary.c.entity_uuid,
+        ),
+    )
+
+
+def _in_current_streak(
+    row: sa.FromClause, boundary: sa.Subquery
+) -> sa.ColumnElement[bool]:
+    """Whether ``row`` is newer than its entity's boundary (or there is none).
+
+    Strictly newer: a row tied with the boundary sits on its resolved side.
+    A boundary proves the object was gone at that instant, and a block that
+    cannot be ordered after the destruction must not seed a new "current"
+    streak — that would mint a survivor exempt from age-out forever for an
+    object that no longer exists.
+    """
+    return sa.or_(boundary.c.boundary.is_(None), row.c.created_on > boundary.c.boundary)
+
+
+def _repeats_an_earlier_block(
+    table: sa.Table, boundary: sa.Subquery
+) -> sa.ColumnElement[bool]:
+    """Whether a same-reason current-streak block precedes this row with no
+    change of reason in between.
+
+    This is the audit writer's suppression rule
+    (:func:`audit.finalize_retention_blocked`) applied retroactively: a
+    block repeating the reason of the block just before it adds nothing,
+    while the first block after a reason change is the only durable record
+    of the new cause and is retained. The comparison is NULL-safe, so
+    consecutive reason-less (pre-feature) rows count as one run, deduped to
+    the run's earliest, and the first coded block ends that run. Note this
+    is *stricter* than the writer for reason-less rows: the writer never
+    suppresses a reason-less block (``_suppress_redundant_block`` bails on a
+    missing code), so here the pruner additionally collapses legacy
+    pre-feature duplicates — the streak's earliest "blocked since" row is
+    still always kept. Tied same-reason rows are not "earlier" than each
+    other, so all of them are kept.
+
+    A ``force`` row is exempt: it is never reported as a repeat (see the
+    return), so it is kept out of the duplicate category and marked a
+    survivor while its streak is current. It is also exempt from operational
+    age-out (see :func:`_operational_candidates`), so an operator force-purge
+    block is retained permanently — never pruned by either category.
+    """
+    earlier: sa.FromClause = table.alias("earlier_block")
+    between: sa.FromClause = table.alias("reason_change")
+    reason_changed_between: sa.ColumnElement[bool] = sa.exists(
+        sa.select(sa.literal(1))
+        .select_from(between)
+        .where(
+            sa.and_(
+                between.c.status == STATUS_BLOCKED,
+                between.c.entity_type == table.c.entity_type,
+                between.c.entity_uuid == table.c.entity_uuid,
+                # Inclusive bounds: a differing-reason block sharing an
+                # exact timestamp with either endpoint still breaks the run,
+                # so a reason-transition row tied with a neighbour is
+                # preserved as a run head rather than pruned as a repeat
+                # (the same preserving-side tie rule the pending and evidence
+                # guards use). Inclusive bounds only ever add boundaries —
+                # i.e. only ever preserve more, never delete more.
+                between.c.created_on >= earlier.c.created_on,
+                between.c.created_on <= table.c.created_on,
+                between.c.reason.is_distinct_from(table.c.reason),
+            )
+        )
+        .correlate(table, earlier)
+    )
+    repeats: sa.ColumnElement[bool] = sa.exists(
+        sa.select(sa.literal(1))
+        .select_from(earlier)
+        .where(
+            sa.and_(
+                earlier.c.status == STATUS_BLOCKED,
+                earlier.c.entity_type == table.c.entity_type,
+                earlier.c.entity_uuid == table.c.entity_uuid,
+                _in_current_streak(earlier, boundary),
+                earlier.c.created_on < table.c.created_on,
+                earlier.c.reason.is_not_distinct_from(table.c.reason),
+                sa.not_(reason_changed_between),
+            )
+        )
+        .correlate(table, boundary)
+    )
+    # A ``force`` attempt is an operator action the writer never suppresses:
+    # audit.py's ``_suppress_redundant_block`` only collapses consecutive
+    # scheduled same-reason blocks, so the pruner does not collapse a force row
+    # either — it is never reported as a repeat. This keeps it out of the
+    # duplicate category and (via ``sa.not_`` in the operational category) marks
+    # it a survivor while its streak is current. It is also exempt from
+    # operational age-out (``_operational_candidates`` excludes force blocks), so
+    # a force block is retained permanently — full immortality for operator
+    # force-purge blocks. A force row may still be the *earlier* anchor a later
+    # scheduled repeat collapses into — only the force row itself is protected
+    # from duplicate removal.
+    return sa.and_(table.c.trigger != TRIGGER_FORCE, repeats)
+
+
+def _preceded_by_unresolved_attempt(table: sa.Table) -> sa.ColumnElement[bool]:
+    """Whether an unresolved (``pending``) attempt precedes this row.
+
+    A ``pending`` row is the only thing that can insert a streak boundary
+    into *history*: every other write lands at ``now``, newer than every
+    existing row, whereas reconciliation and the purge path finalize a
+    pending row **in place**, keeping its original ``created_on``. So a
+    pending row sitting between two blocked rows is a boundary that may
+    appear at any moment, and the blocked row after it would become the new
+    streak's survivor — the very row pruning must never delete.
+
+    A tied timestamp counts as preceding. Which of the two writes landed
+    first is unknowable from the row, the block's classification (current
+    duplicate, or resolved-streak row on an age window) changes with the
+    attempt's outcome, and deferring it until then costs nothing.
+    """
+    pending: sa.FromClause = table.alias("unresolved_attempt")
+    return sa.exists(
+        sa.select(sa.literal(1))
+        .select_from(pending)
+        .where(
+            sa.and_(
+                pending.c.status == STATUS_PENDING,
+                pending.c.entity_type == table.c.entity_type,
+                pending.c.entity_uuid == table.c.entity_uuid,
+                pending.c.created_on <= table.c.created_on,
+            )
+        )
+    )
+
+
+def _duplicate_candidates(now: datetime, limit: int) -> sa.sql.Select:
+    """Select current-streak blocked rows that repeat the block before them.
+
+    Age-independent by design (FR-003): a repeat is prunable the moment the
+    streak holds an earlier same-reason block, regardless of the retention
+    window. Trigger does not gate streak membership — ``scheduled`` and
+    ``force`` blocked rows share streaks — but a ``force`` row is itself never
+    collapsed as a repeat (see :func:`_repeats_an_earlier_block`). Reason *is*
+    a discriminator: the first block after a reason change survives alongside
+    the streak's earliest row.
+
+    Rows preceded by an unresolved attempt are skipped: their classification
+    is not stable, because that attempt can finalize into a boundary and
+    promote them to survivor between selection and deletion. Such rows are
+    collected once the attempt resolves. Note that pruning does not depend on
+    that happening promptly — reconciliation runs from the purge task, which
+    a deployment may have disabled or left in dry-run — so a long-lived
+    pending row defers its successors indefinitely rather than risking them.
+    """
+    table: sa.Table = PurgeAuditLog.__table__
+    boundary: sa.Subquery = _streak_boundary_subquery(now)
+    return (
+        sa.select(table.c.id)
+        .select_from(_with_boundary(table, boundary))
+        .where(table.c.status == STATUS_BLOCKED)
+        .where(table.c.entity_uuid.is_not(None))
+        .where(table.c.created_on <= now)
+        .where(_in_current_streak(table, boundary))
+        .where(_repeats_an_earlier_block(table, boundary))
+        .where(sa.not_(_preceded_by_unresolved_attempt(table)))
+        .order_by(table.c.created_on)
+        .limit(limit)

Review Comment:
   The ID cap bounds the outer re-check, but each row still evaluates the history-sensitive `_repeats_an_earlier_block()` predicates while the coordination lock is held, and `reason` is not covered by either pruning index; a long multi-reason history can therefore keep `write_ahead()` blocked while those probes walk the entity's history. Could we add the promised representative query-plan/runtime evidence or make the locked validation independent of per-entity history?



##########
UPDATING.md:
##########
@@ -243,6 +243,41 @@ unknown impact as zero. Chart and dashboard purge endpoints are unchanged.
 
 - The dashboard datasource-based visibility fallback now fails closed: a dashboard whose member charts’ datasources cannot be resolved (deleted datasource rows, missing `datasource_id`, or unsupported datasource types) is no longer accessible to users without explicit editor/viewer rights, and a dashboard composed of semantic-view charts now requires `datasource_access` on (at least one of) its semantic views or their parent semantic layer — previously any authenticated user could open such a dashboard’s shell. Because the fallback now considers every member chart rather than only table-backed ones, a user holding `datasource_access` on any single member datasource — including a semantic view or its parent layer — can open a mixed dashboard that previously denied them. Dashboards with no charts remain accessible, and dashboards with explicit viewers are unaffected. Conversely, holders of `all_datasource_access` now see every published no-viewer dashboard in the dashboard l
 ist — including chart-less ones previously hidden by the inner joins — matching what the object-level gate already allowed them to open.
 - Version restore (`POST /api/v1/{chart,dashboard,dataset}/<uuid>/versions/<version_uuid>/restore`) now refuses an **externally managed** entity (`is_managed_externally = True`) with HTTP 403, enforcing server-side what the docs already promised. Previously the refusal existed only in the browser, so an otherwise-authorized editor could restore such an entity by calling the endpoint directly and have the restore overwritten on the next external sync. Soft-delete recovery is deliberately unaffected — it changes visibility, not content.
+- The purge audit log can now be pruned automatically. The new
+  `deletion_retention.prune_purge_audit` Celery beat task (daily, 03:30, in the
+  default `CeleryConfig.beat_schedule`) removes duplicate `blocked` records
+  within an entity's current blockage streak (the earliest — "blocked since" —
+  record and the first record after each change of block `reason` always
+  survive, mirroring the audit writer's own suppression rule) and ages out
+  operational records (`blocked` from
+  resolved streaks, `failed`) older than
+  `PURGE_AUDIT_OPERATIONAL_RETENTION_DAYS` (default 90). A streak is ended
+  only by proof the object is gone (`confirmed`/`target_absent`); a `failed`
+  attempt does not reset the "blocked since" record. Completed-destruction
+  evidence (`confirmed`, `target_absent`) is **never touched** unless the
+  separate `PURGE_AUDIT_EVIDENCE_RETENTION_DAYS` opt-in is explicitly set,
+  which is the operator's assertion that an approved compliance policy
+  permits expiring destruction evidence. Automatic deletion is disabled by
+  default; set `PURGE_AUDIT_PRUNING_ENABLED = True` after reviewing these

Review Comment:
   Enabling this in the same rolling deploy can violate the survivor invariant: workers on the base code stamp and commit `write_ahead()` rows without the coordination lock, so one can publish an earlier pending row after the new pruner's locked re-check/delete and later turn that deleted block into the post-boundary survivor. Could the rollout require every audit-writing worker to be on the coordinated protocol before this switch is enabled?



##########
UPDATING.md:
##########
@@ -945,7 +980,7 @@ With the flag on, delete confirmations across the chart/dashboard/dataset list p
 
 This also resolves the limitation noted under *Soft delete and restore for datasets*: a database blocked by soft-deleted datasets can now be freed by purging those datasets (per-entity endpoint, retention task, or `force-purge` CLI) instead of hard-deleting `tables` rows out-of-band.
 
-The `purge_audit_log` table is **never pruned by design** — the audit must survive the entities it names; operators who need to age it out should prune manually.
+Automatic pruning of the `purge_audit_log` table is available but **off by default**: set `PURGE_AUDIT_PRUNING_ENABLED = True` to enable the `deletion_retention.prune_purge_audit` Celery beat task (daily, 03:30) so the table no longer grows unbounded and does not need manual pruning. Left at its default (`PURGE_AUDIT_PRUNING_ENABLED = False`) the table is never pruned and grows indefinitely — enabling it is an explicit operator choice. The policy is written to preserve the audit's meaning rather than trade it away: within an entity's current blockage streak the earliest — "blocked since" — record always survives (only redundant duplicate `blocked` records are collapsed), and completed-destruction evidence (`confirmed`, `target_absent`) is **never** removed unless the separate `PURGE_AUDIT_EVIDENCE_RETENTION_DAYS` opt-in is explicitly set. What ages out is operational noise — `blocked` records from already-resolved streaks and `failed` records — once older than `PURGE_AUDIT
 _OPERATIONAL_RETENTION_DAYS` (default 90). See the release-note entry above for the beat-schedule and `CELERY_CONFIG` details.

Review Comment:
   Enabling pruning still cannot keep this table bounded: every force-triggered `blocked` record is exempt from both duplicate removal and operational expiry, so repeated `force-purge` attempts against a persistently blocked entity grow it forever despite this sentence. Could this document the permanent force-block exception instead of promising that enabling the task makes growth bounded?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org