You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@superset.apache.org by "mikebridge (via GitHub)" <gi...@apache.org> on 2026/09/08 16:50:58 UTC
[PR] fix(versioning): close the two retention-interaction races [superset]
mikebridge opened a new pull request, #44012:
URL: https://github.com/apache/superset/pull/44012
### SUMMARY
Fixes SC-115615 — two retention-interaction races in the versioning backend, both confirmed by two independent review panels (10-lens capstone + Codex capstone Highs #4/#5), sharing test scaffolding. Both scale with `SUPERSET_VERSION_HISTORY_RETENTION_DAYS` set and capture on — the fleet-enablement configuration.
**1. Freshly captured baselines could immediately expire** (`superset/versioning/baseline/insertion.py`). The baseline transaction's `issued_at` was sourced from the entity's historical `changed_on`, but the retention prune keys on `issued_at` — and the baseline's shadow row is closed by the same flush's real edit, so the prune's live-row preservation rule doesn't protect it. Capture a baseline for an entity last edited outside the retention window and the very next prune deleted it: the entity's pre-edit history gone permanently. The baseline is now stamped with **capture time on one clock** (naive-UTC, matching Continuum's storage and the prune's cutoff derivation; the old `sa.func.now()` fallback mixed in the DB server's local clock). Panel chronology is unaffected — baseline-first ordering is by `operation_type`, not `issued_at` — and attribution stays with the pre-versioning author via `user_id`.
**2. Snapshot fetch by OFFSET shifted under a concurrent prune** (`superset/versioning/queries.py`). `get_version` resolved the target to a stable `transaction_id`, threw it away, and re-fetched by positional OFFSET — a prune committing in between shifts the offset and silently surfaces a **different version's snapshot under the requested version uuid**, on the preview/restore read path. The fetch now addresses the row by the resolved `transaction_id` (identity filter kept, so a transaction id recycled on another entity can never match). The restore command already addressed by `transaction_id`; this closes the read side.
### BEFORE/AFTER SCREENSHOTS OR ANIMATED GIF
N/A — backend correctness.
### TESTING INSTRUCTIONS
```bash
python -m pytest tests/unit_tests/versioning -q # 149 passed (4 new in test_retention_races.py)
```
The new tests drive the **real** production functions against a shared Continuum-shaped in-memory store. Reverted-fix control (both fixed files checked out from the base commit): exactly the two race tests flip — `test_baseline_survives_a_full_retention_window_from_capture` (baseline expires immediately) and `test_get_version_returns_resolved_tx_after_concurrent_prune` (OFFSET returns the successor's snapshot) — while the attribution/shadow-shape and no-prune controls stay green in both states (`2 failed, 2 passed`).
### ADDITIONAL INFORMATION
- [x] Has associated issue: SC-115615 (refs apache/superset#41551 capstone; sc-111918 rollout)
- [x] Required feature flags: `ENABLE_VERSIONING_CAPTURE` (+ retention configured, for the code paths involved)
- [ ] Changes UI
- [ ] Includes DB Migration (follow approval process in [SIP-59](https://github.com/apache/superset/issues/13351))
- [ ] Introduces new feature or API
- [ ] Removes existing feature or API
🤖 Generated with [Claude Code](https://claude.com/claude-code)
https://claude.ai/code/session_01JRLEJS4mUqKBoPjSjviKUW
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] fix(versioning): close the two retention-interaction races [superset]
Posted by "rusackas (via GitHub)" <gi...@apache.org>.
rusackas merged PR #44012:
URL: https://github.com/apache/superset/pull/44012
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] fix(versioning): close the two retention-interaction races [superset]
Posted by "bito-code-review[bot] (via GitHub)" <gi...@apache.org>.
bito-code-review[bot] commented on PR #44012:
URL: https://github.com/apache/superset/pull/44012#issuecomment-5641572578
AI Code Review is in progress (usually takes 3 to 15 minutes unless it's a very large PR).
<details>
<summary>
Bito Usage Guide
</summary>
**Commands**
Type the following command in the pull request comment and save the comment.
- `/review` - Manually triggers an incremental AI Review.
- `/review full` - Manually triggers a full AI Review.
- `/pause` - Pauses automatic reviews on this pull request.
- `/resume` - Resumes automatic reviews.
- `/resolve` - Marks all Bito-posted review comments as resolved.
- `/abort` - Cancels all in-progress reviews.
Refer to the <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/available-commands" target="_blank" rel="noopener noreferrer">documentation</a> for additional commands.
**Configuration**
This repository uses `Superset` You can customize the agent settings <a href="https://alpha.bito.ai/home/ai-agents/code-review-agent" target="_blank" rel="noopener noreferrer">here</a> or contact your Bito workspace admin at evan@preset.io.
**Documentation & Help**
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/getting-started/install-run-using-bito-cloud/create-or-customize-an-agent-instance" target="_blank" rel="noopener noreferrer">Customize agent settings</a>
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/implementing-custom-code-review-rules" target="_blank" rel="noopener noreferrer">Review rules</a>
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent" target="_blank" rel="noopener noreferrer">General documentation</a>
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/faqs" target="_blank" rel="noopener noreferrer">FAQ</a>
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] fix(versioning): close the two retention-interaction races [superset]
Posted by "mikebridge (via GitHub)" <gi...@apache.org>.
mikebridge commented on PR #44012:
URL: https://github.com/apache/superset/pull/44012#issuecomment-5589131708
@aminghadersohi — requesting review on the retention-race pair, in your versioning wheelhouse: two data-correctness fixes on the prune path, now through both my four-lens gate and an independent coordinator review (approve; both races confirmed closed under adversarial re-derivation).
**Race 1**: a freshly captured baseline could expire at the very next prune — its transaction was stamped with the entity's historical `changed_on` while retention keys on `issued_at`, and the live-row preservation rule can't save it because the same flush's real edit closes the baseline's shadow row (verified against Continuum's `update_version_validity`). Baselines are now stamped with capture time via a shared `naive_utcnow()` — the *same* helper the prune cutoff derives from, so the one-clock agreement is structural (same UTC reference/derivation; cross-process wall-clock skew is named in the comments and immaterial at day granularity). Attribution stays with the pre-versioning author; ordering is op-type-first so the panel is unaffected.
**Race 2**: `get_version` resolved a stable `transaction_id`, discarded it, and re-fetched by OFFSET — a prune between the two silently served a *different version's snapshot* under the requested uuid. Now fetched by the resolved tx id (shadow PK `(id, transaction_id)` makes the un-ordered `limit(1)` deterministic); the third outcome is pinned — the requested version itself pruned mid-request yields an honest 404, never a neighbour.
Evidence: 6 tests in `test_retention_races.py` drive the real `_insert_baseline_row`, `get_version`, and `_resolve_prune_window` against a Continuum-shaped in-memory store; the reverted-fix control flips exactly the two race tests (the OFFSET failure surfaces as `assert 102 == 101`), and an end-to-end test locks capture-stamp-vs-cutoff through both real callers. One known residual, deliberately out of scope and now traceable as SC-120012 at both sites: the SqlaTable child-snapshot path (`get_version` reads and `revert(relations=…)` *writes* from closed child shadow rows that retention can legitimately prune). Also flagging for your call: `resolve_version_uuid` is now caller-less (documented as retained façade surface) — happy to delete it and the DAO staticmethod in a follow-up if you'd rather remove the prune-unstable index as an attractive nuisance.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] fix(versioning): close the two retention-interaction races [superset]
Posted by "bito-code-review[bot] (via GitHub)" <gi...@apache.org>.
bito-code-review[bot] commented on PR #44012:
URL: https://github.com/apache/superset/pull/44012#issuecomment-5588737616
AI Code Review is in progress (usually takes 3 to 15 minutes unless it's a very large PR).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] fix(versioning): close the two retention-interaction races [superset]
Posted by "mikebridge (via GitHub)" <gi...@apache.org>.
mikebridge commented on PR #44012:
URL: https://github.com/apache/superset/pull/44012#issuecomment-5639914929
Folded in the `naive_utcnow` dedupe as asked (80d3c6683c): the canonical definition stays in `superset/utils/dates.py` and `superset/tasks/utils.py` now re-exports it, so the task modules importing from there are unchanged — behaviour-preserving (both bodies were identical), and the "one helper" docstring claim is now structurally true. Also rebased onto master and refreshed the description's test counts: 6 in the file, the versioning suite is now 158 (master gained 7 versioning tests since `eff38b0`), and the reverted-fix control still flips 4 of 6 — re-run after the dedupe to confirm it can't mask the fix. Thanks for the thorough verification.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] fix(versioning): close the two retention-interaction races [superset]
Posted by "rusackas (via GitHub)" <gi...@apache.org>.
rusackas commented on PR #44012:
URL: https://github.com/apache/superset/pull/44012#issuecomment-5667592001
@gabotorresruiz can your review bot not approve things if you're asking for fixes before merging? Just don't approve, or request changes. Approving something that's not ready can lead to issues if it gets merged before it's _actually_ ready by your review's standards.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] fix(versioning): close the two retention-interaction races [superset]
Posted by "codecov[bot] (via GitHub)" <gi...@apache.org>.
codecov[bot] commented on PR #44012:
URL: https://github.com/apache/superset/pull/44012#issuecomment-5588788387
## [Codecov](https://app.codecov.io/gh/apache/superset/pull/44012?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:x: Patch coverage is `20.00000%` with `4 lines` in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 56.59%. Comparing base ([`7d0cf1a`](https://app.codecov.io/gh/apache/superset/commit/7d0cf1acfc82750b17ca65cb6f6dc1e80efc4f5f?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`6155e8e`](https://app.codecov.io/gh/apache/superset/commit/6155e8ecb444855406f5f6bb2a6cb49d6069e8fe?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
:warning: Report is 1 commits behind head on master.
| [Files with missing lines](https://app.codecov.io/gh/apache/superset/pull/44012?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [superset/versioning/queries.py](https://app.codecov.io/gh/apache/superset/pull/44012?src=pr&el=tree&filepath=superset%2Fversioning%2Fqueries.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-c3VwZXJzZXQvdmVyc2lvbmluZy9xdWVyaWVzLnB5) | 0.00% | [3 Missing :warning: ](https://app.codecov.io/gh/apache/superset/pull/44012?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [superset/versioning/baseline/insertion.py](https://app.codecov.io/gh/apache/superset/pull/44012?src=pr&el=tree&filepath=superset%2Fversioning%2Fbaseline%2Finsertion.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-c3VwZXJzZXQvdmVyc2lvbmluZy9iYXNlbGluZS9pbnNlcnRpb24ucHk=) | 50.00% | [1 Missing :warning: ](https://app.codecov.io/gh/apache/superset/pull/44012?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
> :exclamation: There is a different number of reports uploaded between BASE (7d0cf1a) and HEAD (6155e8e). Click for more details.
>
> <details><summary>HEAD has 5 uploads less than BASE</summary>
>
>| Flag | BASE (7d0cf1a) | HEAD (6155e8e) |
>|------|------|------|
>|python|6|1|
></details>
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #44012 +/- ##
===========================================
- Coverage 79.82% 56.59% -23.23%
===========================================
Files 2914 2914
Lines 170473 170475 +2
Branches 39518 39518
===========================================
- Hits 136076 96477 -39599
- Misses 31875 73510 +41635
+ Partials 2522 488 -2034
```
| [Flag](https://app.codecov.io/gh/apache/superset/pull/44012/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [hive](https://app.codecov.io/gh/apache/superset/pull/44012/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `37.63% <20.00%> (+<0.01%)` | :arrow_up: |
| [mysql](https://app.codecov.io/gh/apache/superset/pull/44012/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [postgres](https://app.codecov.io/gh/apache/superset/pull/44012/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [presto](https://app.codecov.io/gh/apache/superset/pull/44012/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [python](https://app.codecov.io/gh/apache/superset/pull/44012/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `37.63% <20.00%> (-46.57%)` | :arrow_down: |
| [sqlite](https://app.codecov.io/gh/apache/superset/pull/44012/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [unit](https://app.codecov.io/gh/apache/superset/pull/44012/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
</details>
[:umbrella: View full report in Codecov by Harness](https://app.codecov.io/gh/apache/superset/pull/44012?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
- :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] fix(versioning): close the two retention-interaction races [superset]
Posted by "netlify[bot] (via GitHub)" <gi...@apache.org>.
netlify[bot] commented on PR #44012:
URL: https://github.com/apache/superset/pull/44012#issuecomment-5589138592
### <span aria-hidden="true">✅</span> Deploy Preview for *superset-docs-preview* ready!
| Name | Link |
|:-:|------------------------|
|<span aria-hidden="true">🔨</span> Latest commit | eff38b0e3b031b3776f513e98a7b569a089cd48c |
|<span aria-hidden="true">🔍</span> Latest deploy log | https://app.netlify.com/projects/superset-docs-preview/deploys/6aa04437c7ef99000826726e |
|<span aria-hidden="true">😎</span> Deploy Preview | [https://deploy-preview-44012--superset-docs-preview.netlify.app](https://deploy-preview-44012--superset-docs-preview.netlify.app) |
|<span aria-hidden="true">📱</span> Preview on mobile | <details><summary> Toggle QR Code... </summary><br /><br /><br /><br />_Use your smartphone camera to open QR code link._</details> |
|<span aria-hidden="true">🤖</span> Make changes | [Run an agent on this branch](https://app.netlify.com/projects/superset-docs-preview/agent-runs#DcrLDYAgDADQXXqWBJCPuIFjVKjRRIFULsa4u9ze4b1QuVy1wQwwwMqY4959R6GUdcoKpka5HSULxkh3T4nqWZ4l9eYQpTGjj562EKSUk3ZeO4LvBw) |
---
<!-- [superset-docs-preview Preview](https://deploy-preview-44012--superset-docs-preview.netlify.app) -->
_To edit notification comments on pull requests, go to your [Netlify project configuration](https://app.netlify.com/projects/superset-docs-preview/configuration/notifications#deploy-notifications)._
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] fix(versioning): close the two retention-interaction races [superset]
Posted by "bito-code-review[bot] (via GitHub)" <gi...@apache.org>.
bito-code-review[bot] commented on PR #44012:
URL: https://github.com/apache/superset/pull/44012#issuecomment-5594521111
AI Code Review is in progress (usually takes 3 to 15 minutes unless it's a very large PR).
<details>
<summary>
Bito Usage Guide
</summary>
**Commands**
Type the following command in the pull request comment and save the comment.
- `/review` - Manually triggers an incremental AI Review.
- `/review full` - Manually triggers a full AI Review.
- `/pause` - Pauses automatic reviews on this pull request.
- `/resume` - Resumes automatic reviews.
- `/resolve` - Marks all Bito-posted review comments as resolved.
- `/abort` - Cancels all in-progress reviews.
Refer to the <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/available-commands" target="_blank" rel="noopener noreferrer">documentation</a> for additional commands.
**Configuration**
This repository uses `Superset` You can customize the agent settings <a href="https://alpha.bito.ai/home/ai-agents/code-review-agent" target="_blank" rel="noopener noreferrer">here</a> or contact your Bito workspace admin at evan@preset.io.
**Documentation & Help**
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/getting-started/install-run-using-bito-cloud/create-or-customize-an-agent-instance" target="_blank" rel="noopener noreferrer">Customize agent settings</a>
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/implementing-custom-code-review-rules" target="_blank" rel="noopener noreferrer">Review rules</a>
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent" target="_blank" rel="noopener noreferrer">General documentation</a>
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/faqs" target="_blank" rel="noopener noreferrer">FAQ</a>
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org