You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@superset.apache.org by "mikebridge (via GitHub)" <gi...@apache.org> on 2026/08/24 19:21:56 UTC
[PR] feat(deletion-retention): persist block/failure reason codes on the purge audit log [superset]
mikebridge opened a new pull request, #43485:
URL: https://github.com/apache/superset/pull/43485
### SUMMARY
Blocked purge audit records currently record only a status: an auditor reading `purge_audit_log` cannot distinguish "blocked because an alert/report references the entity" from "blocked because a user has the dashboard as their welcome page" from "failed on a restrictive foreign key the cascade does not handle" — outcomes with entirely different remediations. The reason is already known at decision time (logs, CLI output) but is discarded before it reaches the durable record.
This PR threads a **stable machine-readable reason code** from the purge decision points into the audit record, and amends the #42863 growth-suppression predicate to key on **status and reason**:
- **Vocabulary**: `REASON_REPORT_SCHEDULE` (`"report_schedule"`), `REASON_USER_ATTRIBUTE` (`"user_attribute"`), and `REASON_UNHANDLED_REFERENCE` (`"unhandled_reference"`) declared in `purge_policy.py` (the vocabulary's owner; deliberately model-free). The values are frozen identifiers pinned by a golden-set test — a future physical table rename must not re-mint persisted history. `PurgeBlockedError` carries the code alongside its human phrase; the first declared blocker that matches wins (declaration order is part of the contract, unit- and integration-tested).
- **Threading**: `CascadeResult.blocked_reason_code` set by both cascade handlers (policy block from the exception; `IntegrityError` → `unhandled_reference`); `audit.block()` and `audit.finalize_retention_blocked()` take the reason as a required parameter, carried through the retain path and **both** crash-recovery branches as a parameter (never read from the reason-less pending row).
- **Reading rule**: an unhandled-FK failure keeps `blocked` status (so growth suppression continues to apply) — `status = blocked AND reason = unhandled_reference` denotes a cascade-coverage defect, not an intended policy outcome. Run-level metrics conflate this class with reference blocks, making the persisted code the sole durable classifier.
- **Suppression amendment**: same-status-same-reason nights stay suppressed; a reason change writes exactly one new blocked row carrying the new code. The predecessor is the **latest** same-entity retention row overall (clock skew or an overlapping run can no longer suppress against a non-latest row — the newer row fails the strictly-older check and the attempt is retained, fail-safe). NULL-safe both ways: a reason-less (pre-upgrade) predecessor never matches, and a missing current code never suppresses (WARNING, fail-open). Timestamp-tie ambiguity widens to "differs in status **or** reason" via `is_distinct_from` (NULL-safe on PostgreSQL/MySQL/SQLite).
- **Strictly reason-less**: failed, confirmed, race-lost, and reconciled rows never carry a reason. The known crash window (worker dies between the blocked decision and finalization; `reconcile_pending` later finalizes failed/NULL) is documented and deliberately not "fixed" by fabricating a code — the next night re-anchors with the real one.
- **Migration**: one additive nullable `String(64)` column via the shared migration utils. No backfill, no index, no reader changes; historical rows keep NULL and every reader tolerates absence. The #42863 suppression paragraph in `UPDATING.md` is amended to state the new predicate.
### BEFORE/AFTER SCREENSHOTS OR ANIMATED GIF
Backend-only. Before/after of the audit table for the same three blocked outcomes:
```
-- BEFORE
status | reason
blocked | (column does not exist)
blocked |
blocked |
-- AFTER
status | reason
blocked | report_schedule
blocked | user_attribute
blocked | unhandled_reference
```
### TESTING INSTRUCTIONS
```bash
# unit: vocabulary golden set, threading, first-match
pytest tests/unit_tests/commands/deletion_retention/ tests/unit_tests/tasks/test_deletion_retention.py
# integration: three-way distinction, suppression contract incl. reason-change,
# NULL-safety, latest-predecessor ordering, recovery branches, immutability
pytest tests/integration_tests/deletion_retention/
# migration round-trip (verified on SQLite and PostgreSQL)
superset db upgrade && superset db downgrade 1072de5ed955 && superset db upgrade
```
Manual: soft-delete a chart referenced by an active report, run the retention purge (or `superset deletion-retention force-purge`), then `SELECT status, reason FROM purge_audit_log` — the blocked row carries `report_schedule`.
The changed-reason suppression test was control-run: it fails against the pre-amendment status-only predicate and passes with this PR.
### ADDITIONAL INFORMATION
- [ ] Has associated issue:
- [ ] Required feature flags:
- [ ] Changes UI
- [x] Includes DB Migration (follow approval process in [SIP-59](https://github.com/apache/superset/issues/13351))
- [x] Migration is atomic, supports rollback & is backwards-compatible
- [x] Confirm DB migration upgrade and downgrade tested
- [x] Runtime estimates and downtime expectations provided
- [ ] Introduces new feature or API
- [ ] Removes existing feature or API
Runtime: metadata-only DDL on PostgreSQL/SQLite; instant DDL on MySQL 8.0.12+ (trailing nullable ADD COLUMN); table rebuild only on older MySQL — `purge_audit_log` is small and growth-suppressed by design. No downtime expected. Single alembic head verified (`1072de5ed955`) before authoring.
Internal tracking: sc-115342. Follows #42863 (audit dedupe) and #42888 (declarative purge policies).
*This PR was generated by Claude (AI) on behalf of @mikebridge.*
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(deletion-retention): persist block/failure reason codes on the purge audit log [superset]
Posted by "codecov[bot] (via GitHub)" <gi...@apache.org>.
codecov[bot] commented on PR #43485:
URL: https://github.com/apache/superset/pull/43485#issuecomment-5400384419
## [Codecov](https://app.codecov.io/gh/apache/superset/pull/43485?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:x: Patch coverage is `92.68293%` with `3 lines` in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 66.02%. Comparing base ([`8fa48d7`](https://app.codecov.io/gh/apache/superset/commit/8fa48d73f6938d98623c1471406a64a218fd9a4b?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`2b0b72a`](https://app.codecov.io/gh/apache/superset/commit/2b0b72a414a2c45ce2eb87b8a8056ffd83d2993f?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
:warning: Report is 33 commits behind head on master.
| [Files with missing lines](https://app.codecov.io/gh/apache/superset/pull/43485?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [superset/tasks/deletion\_retention.py](https://app.codecov.io/gh/apache/superset/pull/43485?src=pr&el=tree&filepath=superset%2Ftasks%2Fdeletion_retention.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-c3VwZXJzZXQvdGFza3MvZGVsZXRpb25fcmV0ZW50aW9uLnB5) | 80.00% | [1 Missing and 1 partial :warning: ](https://app.codecov.io/gh/apache/superset/pull/43485?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...perset/commands/deletion\_retention/purge\_policy.py](https://app.codecov.io/gh/apache/superset/pull/43485?src=pr&el=tree&filepath=superset%2Fcommands%2Fdeletion_retention%2Fpurge_policy.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-c3VwZXJzZXQvY29tbWFuZHMvZGVsZXRpb25fcmV0ZW50aW9uL3B1cmdlX3BvbGljeS5weQ==) | 90.00% | [0 Missing and 1 partial :warning: ](https://app.codecov.io/gh/apache/superset/pull/43485?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #43485 +/- ##
===========================================
- Coverage 78.85% 66.02% -12.83%
===========================================
Files 2876 2877 +1
Lines 164601 165000 +399
Branches 38015 38086 +71
===========================================
- Hits 129799 108949 -20850
- Misses 32355 53829 +21474
+ Partials 2447 2222 -225
```
| [Flag](https://app.codecov.io/gh/apache/superset/pull/43485/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [hive](https://app.codecov.io/gh/apache/superset/pull/43485/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `38.02% <34.14%> (-0.06%)` | :arrow_down: |
| [mysql](https://app.codecov.io/gh/apache/superset/pull/43485/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [postgres](https://app.codecov.io/gh/apache/superset/pull/43485/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [presto](https://app.codecov.io/gh/apache/superset/pull/43485/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `39.94% <34.14%> (-0.06%)` | :arrow_down: |
| [python](https://app.codecov.io/gh/apache/superset/pull/43485/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `57.63% <92.68%> (-25.92%)` | :arrow_down: |
| [sqlite](https://app.codecov.io/gh/apache/superset/pull/43485/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `57.49% <92.68%> (-0.03%)` | :arrow_down: |
| [unit](https://app.codecov.io/gh/apache/superset/pull/43485/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
</details>
[:umbrella: View full report in Codecov by Harness](https://app.codecov.io/gh/apache/superset/pull/43485?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
- :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(deletion-retention): persist purge block reason codes on the audit log [superset]
Posted by "rusackas (via GitHub)" <gi...@apache.org>.
rusackas merged PR #43485:
URL: https://github.com/apache/superset/pull/43485
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(deletion-retention): persist purge block reason codes on the audit log [superset]
Posted by "bito-code-review[bot] (via GitHub)" <gi...@apache.org>.
bito-code-review[bot] commented on PR #43485:
URL: https://github.com/apache/superset/pull/43485#issuecomment-5402749552
AI Code Review is in progress (usually takes 3 to 15 minutes unless it's a very large PR).
<details>
<summary>
Bito Usage Guide
</summary>
**Commands**
Type the following command in the pull request comment and save the comment.
- `/review` - Manually triggers an incremental AI Review.
- `/review full` - Manually triggers a full AI Review.
- `/pause` - Pauses automatic reviews on this pull request.
- `/resume` - Resumes automatic reviews.
- `/resolve` - Marks all Bito-posted review comments as resolved.
- `/abort` - Cancels all in-progress reviews.
Refer to the <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/available-commands" target="_blank" rel="noopener noreferrer">documentation</a> for additional commands.
**Configuration**
This repository uses `Superset` You can customize the agent settings <a href="https://alpha.bito.ai/home/ai-agents/code-review-agent" target="_blank" rel="noopener noreferrer">here</a> or contact your Bito workspace admin at evan@preset.io.
**Documentation & Help**
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/getting-started/install-run-using-bito-cloud/create-or-customize-an-agent-instance" target="_blank" rel="noopener noreferrer">Customize agent settings</a>
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/implementing-custom-code-review-rules" target="_blank" rel="noopener noreferrer">Review rules</a>
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent" target="_blank" rel="noopener noreferrer">General documentation</a>
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/faqs" target="_blank" rel="noopener noreferrer">FAQ</a>
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(deletion-retention): persist block/failure reason codes on the purge audit log [superset]
Posted by "netlify[bot] (via GitHub)" <gi...@apache.org>.
netlify[bot] commented on PR #43485:
URL: https://github.com/apache/superset/pull/43485#issuecomment-5400194342
### <span aria-hidden="true">✅</span> Deploy Preview for *superset-docs-preview* ready!
| Name | Link |
|:-:|------------------------|
|<span aria-hidden="true">🔨</span> Latest commit | 2b0b72a414a2c45ce2eb87b8a8056ffd83d2993f |
|<span aria-hidden="true">🔍</span> Latest deploy log | https://app.netlify.com/projects/superset-docs-preview/deploys/6a8c99d4153dc500086008c9 |
|<span aria-hidden="true">😎</span> Deploy Preview | [https://deploy-preview-43485--superset-docs-preview.netlify.app](https://deploy-preview-43485--superset-docs-preview.netlify.app) |
|<span aria-hidden="true">📱</span> Preview on mobile | <details><summary> Toggle QR Code... </summary><br /><br /><br /><br />_Use your smartphone camera to open QR code link._</details> |
|<span aria-hidden="true">🤖</span> Make changes | [Run an agent on this branch](https://app.netlify.com/projects/superset-docs-preview/agent-runs#DcpBCoAgEEDRu8w6QUtFu0HH0FEKska0FhHdvVl8-Iv3Qm101AtmgAFiCydu_B2FUmbSo6h3W7OIhXAXLYdOZ2eYci30LImpDQ69T5p5QiOldJZDD98P) |
---
<!-- [superset-docs-preview Preview](https://deploy-preview-43485--superset-docs-preview.netlify.app) -->
_To edit notification comments on pull requests, go to your [Netlify project configuration](https://app.netlify.com/projects/superset-docs-preview/configuration/notifications#deploy-notifications)._
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(deletion-retention): persist block/failure reason codes on the purge audit log [superset]
Posted by "bito-code-review[bot] (via GitHub)" <gi...@apache.org>.
bito-code-review[bot] commented on PR #43485:
URL: https://github.com/apache/superset/pull/43485#issuecomment-5400174300
AI Code Review is in progress (usually takes 3 to 15 minutes unless it's a very large PR).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(deletion-retention): persist purge block reason codes on the audit log [superset]
Posted by "bito-code-review[bot] (via GitHub)" <gi...@apache.org>.
bito-code-review[bot] commented on PR #43485:
URL: https://github.com/apache/superset/pull/43485#issuecomment-5415845307
**Bito Automatic Review Skipped – PR Already Merged**<br/><br/>Bito scheduled an automatic review for this pull request, but the review was skipped because this PR was merged before the review could be run.<br/>No action is needed if you didn't intend to review it. To get a review, you can type `/review` in a comment and save it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org