You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@superset.apache.org by "alexandrusoare (via GitHub)" <gi...@apache.org> on 2026/10/02 11:16:05 UTC
[PR] feat(retention): let a host install purge policies for its own soft-delete roots [superset]
alexandrusoare opened a new pull request, #44892:
URL: https://github.com/apache/superset/pull/44892
### SUMMARY
The scheduled purge discovers what to purge through the `SoftDeleteMixin` subclass registry, which any model adopting the mixin joins automatically — including models belonging to a host distribution built on Superset. The purge policy registry, by contrast, is a closed dict of three roots, so such a model is discovered and then cannot be purged. #44874 made that case report itself instead of failing per row; this adds the way out of it.
`purge_policy_registry()` is split into the built-in declarations, unchanged and still built once per process, and an index over those plus any policies a host installed under the new `PURGE_POLICIES_FUNC` config key. Installed policies are validated against the discovered dependency graph exactly as the built-in ones are, so a declaration that doesn't match the real foreign keys is rejected rather than used to delete rows.
### BEFORE/AFTER SCREENSHOTS OR ANIMATED GIF
<!--- Skip this if not applicable -->
### TESTING INSTRUCTIONS
<!--- Required! What steps can be taken to manually verify the changes? -->
### ADDITIONAL INFORMATION
<!--- Check any relevant boxes with "x" -->
<!--- HINT: Include "Fixes #nnn" if you are fixing an existing issue -->
- [ ] Has associated issue:
- [ ] Required feature flags:
- [ ] Changes UI
- [ ] Includes DB Migration (follow approval process in [SIP-59](https://github.com/apache/superset/issues/13351))
- [ ] Migration is atomic, supports rollback & is backwards-compatible
- [ ] Confirm DB migration upgrade and downgrade tested
- [ ] Runtime estimates and downtime expectations provided
- [ ] Introduces new feature or API
- [ ] Removes existing feature or API
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(retention): let a host install purge policies for its own soft-delete roots [superset]
Posted by "sadpandajoe (via GitHub)" <gi...@apache.org>.
sadpandajoe commented on code in PR #44892:
URL: https://github.com/apache/superset/pull/44892#discussion_r4165464820
##########
docs/docs/using-superset/recently-archived.mdx:
##########
@@ -123,7 +123,18 @@ Each skip increments `deletion_retention.unsupported_models.<table>` once per
run. Unsupported models are excluded from `would_purge` and do not count as
`cascade_failures`; supported models continue to use their existing policies.
An empty `unsupported_models` map means no registered models were skipped for
-this reason. This reporting does not add purge support for host-defined models.
+this reason. This reporting does not itself add purge support for host-defined
+models.
+
+A host distribution that carries its own soft-deletable entities can supply
+their purge behavior by installing `PURGE_POLICIES_FUNC`: a callable returning
+one `PurgeEntityPolicy` per root it owns. Installed policies are validated
+against the discovered dependency graph exactly as the built-in ones are, and a
+model that gains an installed policy stops being reported as unsupported. A
+provider that fails, returns something other than a sequence of policies, or
+redeclares a built-in chart, dashboard or dataset root is logged and ignored,
+leaving the built-in roots unchanged. Hosts can require
+`superset.versioning.utils.HOST_POLICY_API_VERSION` before installing.
Review Comment:
`HOST_POLICY_API_VERSION` is still `1` on the base branch, where `PURGE_POLICIES_FUNC` is never read, so a host using this guard can pass startup while its archived roots remain unsupported after a rollback. Could this document a capability check or required version that actually distinguishes purge-hook support?
##########
superset/commands/deletion_retention/purge_policy.py:
##########
@@ -860,7 +867,96 @@ def declare(
cleanup_permission=cleanup_dataset_permission,
),
}
- return validate_unique_root_policies(registry.values())
+ return tuple(registry.values())
+
+
+def _host_purge_policies() -> tuple[PurgeEntityPolicy, ...]:
+ """Return the purge policies a host installed for its own roots.
+
+ A host distribution can carry ``SoftDeleteMixin`` entities this package
+ cannot import. The retention task discovers those entities through the
+ mixin registry, so without a policy they reach the cascade as an
+ unsupported model. The host therefore declares their purge behavior and
+ installs it under ``PURGE_POLICIES_FUNC``.
+
+ Host boundary: an unavailable provider, a malformed payload, a policy that
+ collides with a root declared here, and two policies for one host root are
+ each logged and dropped. A broken host declaration must not take the
+ scheduled purge down with it, and must never redefine how a chart,
+ dashboard or dataset is purged.
+ """
+ if not has_app_context():
+ return ()
+ provider: Callable[[], Any] | None = current_app.config.get(
+ HOST_POLICIES_CONFIG_KEY
+ )
+ if provider is None:
+ return ()
+ try:
+ provided: Any = provider()
+ except Exception: # pylint: disable=broad-except
+ logger.exception(
+ "purge_policy: %s is unavailable; keeping built-in roots only",
+ HOST_POLICIES_CONFIG_KEY,
+ )
+ return ()
+ if not isinstance(provided, (list, tuple)) or not all(
+ isinstance(policy, PurgeEntityPolicy) for policy in provided
+ ):
+ logger.error(
+ "purge_policy: %s returned %s; expected a sequence of PurgeEntityPolicy",
+ HOST_POLICIES_CONFIG_KEY,
+ type(provided).__name__,
+ )
+ return ()
+ builtin_roots: frozenset[type[Any]] = frozenset(
+ policy.model for policy in _builtin_purge_policies()
+ )
+ declared: dict[type[Any], int] = {}
+ for policy in provided:
+ declared[policy.model] = declared.get(policy.model, 0) + 1
+ # Which of two declarations for one root is authoritative is undecidable,
+ # and the loser would still delete rows. Dropping both leaves the model
+ # reported as unsupported, which is the recoverable outcome. Resolving it
+ # here also keeps the duplicate away from validate_unique_root_policies,
+ # whose ValueError would abort the whole scheduled run.
+ duplicated: list[type[Any]] = [
+ model for model, count in declared.items() if count > 1
+ ]
+ for model in duplicated:
+ logger.error(
+ "purge_policy: %s declares %d policies for %s; ignoring all of them",
+ HOST_POLICIES_CONFIG_KEY,
+ declared[model],
+ model.__name__,
+ )
+ accepted: list[PurgeEntityPolicy] = []
+ for policy in provided:
+ if policy.model in builtin_roots:
+ logger.error(
+ "purge_policy: host policy for built-in root %s ignored",
+ policy.model.__name__,
+ )
+ continue
+ if policy.model in duplicated:
+ continue
+ accepted.append(policy)
+ return tuple(accepted)
+
+
+def purge_policy_registry() -> Mapping[type[Any], PurgeEntityPolicy]:
+ """Index the built-in purge roots plus any the host installed.
+
+ Uncached on purpose, unlike its two inputs: the built-in declarations are
+ built once per process, while a host policy is resolved per call, so a
+ provider installed after the first purge is honored for any root not yet
+ resolved. ``get_purge_policy`` memoizes per model, so replacing the policy
+ of a root it has already resolved needs a restart -- and that memoization
+ is why rebuilding this small index is off the hot path.
+ """
+ return validate_unique_root_policies(
Review Comment:
A host graph classified as root → ASSOCIATION link → OWNED detail passes validation with the stock cleanup callbacks, but the association phase deletes the link before the detail query traverses it; SQLite without FK enforcement then leaves the detail orphaned, while enforced FKs make every purge roll back. Could validation reject this mixed-phase ordering, or could cleanup preserve the traversal keys until the descendants are deleted?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(retention): let a host install purge policies for its own soft-delete roots [superset]
Posted by "alexandrusoare (via GitHub)" <gi...@apache.org>.
alexandrusoare merged PR #44892:
URL: https://github.com/apache/superset/pull/44892
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(retention): let a host install purge policies for its own soft-delete roots [superset]
Posted by "bito-code-review[bot] (via GitHub)" <gi...@apache.org>.
bito-code-review[bot] commented on code in PR #44892:
URL: https://github.com/apache/superset/pull/44892#discussion_r4187287476
##########
tests/unit_tests/tasks/test_deletion_retention.py:
##########
@@ -330,6 +331,79 @@ class UnsupportedModel(SoftDeleteMixin):
engine.dispose()
+def test_scan_failure_keeps_the_counts_earned_before_it(app_context: None) -> None:
+ """A scan that fails part-way reports itself and keeps what it purged.
+
+ By the time a later page fails, the earlier page's deletions are
+ committed. Discarding the counts would make the run's summary and its
+ purge gauges understate what was actually removed.
+ """
+ import superset.tasks.deletion_retention as mod
+ from superset.commands.deletion_retention.purge_cascade import CascadeResult
+ from superset.models.slice import Slice
+
+ purged_result: CascadeResult = CascadeResult(
+ purged=True, entity_type="chart", entity_uuid="gone"
+ )
+
+ def pages(*args: Any, **kwargs: Any) -> Iterator[list[int]]:
+ yield [1, 2]
+ raise RuntimeError("no such column: id")
+
+ with (
+ patch.object(mod, "_iter_eligible_ids", side_effect=pages),
+ patch.object(mod, "_purge_one", return_value=purged_result),
+ ):
+ purged, would, failures, blocked, scan_failures = mod._purge_model(
+ Slice, datetime.now(), dry_run=False
+ )
+
+ assert (purged, would, failures, blocked) == (2, 0, 0, 0)
+ assert scan_failures == 1
+
+
+def test_root_without_a_table_name_does_not_abort_the_run(
+ app_config: Config,
+ monkeypatch: pytest.MonkeyPatch,
+) -> None:
+ """Resolving a root's table name is itself guarded.
+
+ The name is read off the model, so a root that cannot supply one must be
+ counted and skipped like any other failing root -- not end the pass before
+ the roots that could have purged are reached.
+ """
+ # avoid app-init regression: model helpers require the app_config fixture first.
+ from superset.models.helpers import SoftDeleteMixin
+ from superset.tasks import deletion_retention as task
+
+ supported_models: list[type[SoftDeleteMixin]] = list(task.purge_policy_registry())
+
+ class NoTableName(SoftDeleteMixin):
+ """A registered root whose table name cannot be read."""
+
+ monkeypatch.setattr(
+ SoftDeleteMixin,
+ "_registered_subclasses",
+ [NoTableName, *supported_models],
+ )
Review Comment:
<div>
<div id="suggestion">
<div id="issue"><b>Registry pollution leaks across tests</b></div>
<div id="fix">
`NoTableName` leaks into the real `SoftDeleteMixin._registered_subclasses`: `__init_subclass__` (superset/models/helpers.py:1374-1382) appends it to the live list at class-definition time (line 381), before `monkeypatch.setattr` (384) rebinds the attribute — and setattr restores the original list object, still containing `NoTableName`. Sibling tests avoid this by swapping the registry before defining the class (lines 256→259, 420→422). The leaked unmapped, table-less class is consumed by the visibility listener via `_all_soft_delete_subclasses` (helpers.py:1489) and by `_soft_delete_models()` in later tests.
</div>
</div>
<small><i>Code Review Run #7e6878</i></small>
</div>
---
Should Bito avoid suggestions like this for future reviews? (<a href=https://alpha.bito.ai/home/ai-agents/review-rules>Manage Rules</a>)
- [ ] Yes, avoid them
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(retention): let a host install purge policies for its own soft-delete roots [superset]
Posted by "codeant-ai-for-open-source[bot] (via GitHub)" <gi...@apache.org>.
codeant-ai-for-open-source[bot] commented on code in PR #44892:
URL: https://github.com/apache/superset/pull/44892#discussion_r4165236784
##########
superset/commands/deletion_retention/purge_policy.py:
##########
@@ -860,7 +867,74 @@ def declare(
cleanup_permission=cleanup_dataset_permission,
),
}
- return validate_unique_root_policies(registry.values())
+ return tuple(registry.values())
+
+
+def _host_purge_policies() -> tuple[PurgeEntityPolicy, ...]:
+ """Return the purge policies a host installed for its own roots.
+
+ A host distribution can carry ``SoftDeleteMixin`` entities this package
+ cannot import. The retention task discovers those entities through the
+ mixin registry, so without a policy they reach the cascade as an
+ unsupported model. The host therefore declares their purge behavior and
+ installs it under ``PURGE_POLICIES_FUNC``.
+
+ Host boundary: an unavailable provider, a malformed payload, or a policy
+ that collides with a root declared here is logged and dropped. A broken
+ host declaration must not take the scheduled purge down with it, and must
+ never redefine how a chart, dashboard or dataset is purged.
+ """
+ if not has_app_context():
+ return ()
+ provider: Callable[[], Any] | None = current_app.config.get(
+ HOST_POLICIES_CONFIG_KEY
+ )
+ if provider is None:
+ return ()
+ try:
+ provided: Any = provider()
+ except Exception: # pylint: disable=broad-except
+ logger.exception(
+ "purge_policy: %s is unavailable; keeping built-in roots only",
+ HOST_POLICIES_CONFIG_KEY,
+ )
+ return ()
+ if not isinstance(provided, (list, tuple)) or not all(
+ isinstance(policy, PurgeEntityPolicy) for policy in provided
+ ):
+ logger.error(
+ "purge_policy: %s returned %s; expected a sequence of PurgeEntityPolicy",
+ HOST_POLICIES_CONFIG_KEY,
+ type(provided).__name__,
+ )
+ return ()
+ builtin_roots: frozenset[type[Any]] = frozenset(
+ policy.model for policy in _builtin_purge_policies()
+ )
+ accepted: list[PurgeEntityPolicy] = []
+ for policy in provided:
+ if policy.model in builtin_roots:
+ logger.error(
+ "purge_policy: host policy for built-in root %s ignored",
+ policy.model.__name__,
+ )
+ continue
+ accepted.append(policy)
+ return tuple(accepted)
Review Comment:
**Suggestion:** Duplicate policies for the same non-built-in model reach `validate_unique_root_policies`, which raises and aborts the scheduled purge instead of ignoring the broken host declaration.
**Assessment:** 🟠 `Major` · 🔁 `Occurrence: Rarely` · 🏷️ `Logic error`
[](https://docs.codeant.ai/cli/resolve-pr-comments-skill) [](https://app.codeant.ai/fix-in-ide?tool=cursor&prompt_id=808969e2817b461da13c5cb54c45ca1c&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset) [](https://app.codeant.ai/fix-in-ide?tool=vscode-claude&prompt_id=808969e2817b461da13c5cb54c45ca1c&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset)
<details>
<summary><b>Prompt for AI Agent 🤖 </b></summary>
```mdx
This is a comment left during a code review.
**Path:** superset/commands/deletion_retention/purge_policy.py
**Line:** 922:923
**Comment:**
*Logic Error: Duplicate policies for the same non-built-in model reach `validate_unique_root_policies`, which raises and aborts the scheduled purge instead of ignoring the broken host declaration.
Validate the correctness of the flagged issue. If correct, How can I resolve this? If you propose a fix, implement it and please make it concise.
Once fix is implemented, also check other comments on the same PR, and ask user if the user wants to fix the rest of the comments as well. if said yes, then fetch all the comments validate the correctness and implement a minimal fix
```
</details>
<a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F44892&comment_hash=4590511b397b26bbbd9d8ed38649f93887d2a3558e85b8596112ae006a3d8ec7&reaction=like'>👍</a> | <a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F44892&comment_hash=4590511b397b26bbbd9d8ed38649f93887d2a3558e85b8596112ae006a3d8ec7&reaction=dislike'>👎</a>
##########
superset/commands/deletion_retention/purge_policy.py:
##########
@@ -860,7 +867,74 @@ def declare(
cleanup_permission=cleanup_dataset_permission,
),
}
- return validate_unique_root_policies(registry.values())
+ return tuple(registry.values())
+
+
+def _host_purge_policies() -> tuple[PurgeEntityPolicy, ...]:
+ """Return the purge policies a host installed for its own roots.
+
+ A host distribution can carry ``SoftDeleteMixin`` entities this package
+ cannot import. The retention task discovers those entities through the
+ mixin registry, so without a policy they reach the cascade as an
+ unsupported model. The host therefore declares their purge behavior and
+ installs it under ``PURGE_POLICIES_FUNC``.
+
+ Host boundary: an unavailable provider, a malformed payload, or a policy
+ that collides with a root declared here is logged and dropped. A broken
+ host declaration must not take the scheduled purge down with it, and must
+ never redefine how a chart, dashboard or dataset is purged.
+ """
+ if not has_app_context():
+ return ()
+ provider: Callable[[], Any] | None = current_app.config.get(
+ HOST_POLICIES_CONFIG_KEY
+ )
+ if provider is None:
+ return ()
+ try:
+ provided: Any = provider()
+ except Exception: # pylint: disable=broad-except
+ logger.exception(
+ "purge_policy: %s is unavailable; keeping built-in roots only",
+ HOST_POLICIES_CONFIG_KEY,
+ )
+ return ()
+ if not isinstance(provided, (list, tuple)) or not all(
+ isinstance(policy, PurgeEntityPolicy) for policy in provided
+ ):
+ logger.error(
+ "purge_policy: %s returned %s; expected a sequence of PurgeEntityPolicy",
+ HOST_POLICIES_CONFIG_KEY,
+ type(provided).__name__,
+ )
+ return ()
+ builtin_roots: frozenset[type[Any]] = frozenset(
+ policy.model for policy in _builtin_purge_policies()
+ )
+ accepted: list[PurgeEntityPolicy] = []
+ for policy in provided:
+ if policy.model in builtin_roots:
+ logger.error(
+ "purge_policy: host policy for built-in root %s ignored",
+ policy.model.__name__,
+ )
+ continue
+ accepted.append(policy)
+ return tuple(accepted)
+
+
+def purge_policy_registry() -> Mapping[type[Any], PurgeEntityPolicy]:
+ """Index the built-in purge roots plus any the host installed.
+
+ Uncached on purpose, unlike its two inputs: the built-in declarations are
+ built once per process, while a host policy is resolved per call so a
+ provider installed after the first purge is still honored. The hot path is
+ ``get_purge_policy``, which caches per model, so rebuilding this small
+ index is not on it.
+ """
+ return validate_unique_root_policies(
+ (*_builtin_purge_policies(), *_host_purge_policies())
+ )
Review Comment:
**Suggestion:** The cached resolver can keep an earlier host policy after the provider changes, so newly installed or replacement policies are not honored by `get_purge_policy`.
**Assessment:** 🟠 `Major` · 🔁 `Occurrence: Rarely` · 🏷️ `Cache`
[](https://docs.codeant.ai/cli/resolve-pr-comments-skill) [](https://app.codeant.ai/fix-in-ide?tool=cursor&prompt_id=27b05162e10c47019893d9862974becf&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset) [](https://app.codeant.ai/fix-in-ide?tool=vscode-claude&prompt_id=27b05162e10c47019893d9862974becf&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset)
<details>
<summary><b>Prompt for AI Agent 🤖 </b></summary>
```mdx
This is a comment left during a code review.
**Path:** superset/commands/deletion_retention/purge_policy.py
**Line:** 935:937
**Comment:**
*Cache: The cached resolver can keep an earlier host policy after the provider changes, so newly installed or replacement policies are not honored by `get_purge_policy`.
Validate the correctness of the flagged issue. If correct, How can I resolve this? If you propose a fix, implement it and please make it concise.
Once fix is implemented, also check other comments on the same PR, and ask user if the user wants to fix the rest of the comments as well. if said yes, then fetch all the comments validate the correctness and implement a minimal fix
```
</details>
<a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F44892&comment_hash=11f9bf4c02ff008c3a6869668fa6dabf2ef9f4096ac0b56a61e5b49703340462&reaction=like'>👍</a> | <a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F44892&comment_hash=11f9bf4c02ff008c3a6869668fa6dabf2ef9f4096ac0b56a61e5b49703340462&reaction=dislike'>👎</a>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(retention): let a host install purge policies for its own soft-delete roots [superset]
Posted by "netlify[bot] (via GitHub)" <gi...@apache.org>.
netlify[bot] commented on PR #44892:
URL: https://github.com/apache/superset/pull/44892#issuecomment-5951190855
### <span aria-hidden="true">✅</span> Deploy Preview for *superset-docs-preview* ready!
| Name | Link |
|:-:|------------------------|
|<span aria-hidden="true">🔨</span> Latest commit | ee7adb9414d9cbf9cde8e208b01d95f654b48175 |
|<span aria-hidden="true">🔍</span> Latest deploy log | https://app.netlify.com/projects/superset-docs-preview/deploys/6abf92768a1a1a0008618814 |
|<span aria-hidden="true">😎</span> Deploy Preview | [https://deploy-preview-44892--superset-docs-preview.netlify.app](https://deploy-preview-44892--superset-docs-preview.netlify.app) |
|<span aria-hidden="true">📱</span> Preview on mobile | <details><summary> Toggle QR Code... </summary><br /><br /><br /><br />_Use your smartphone camera to open QR code link._</details> |
|<span aria-hidden="true">🤖</span> Make changes | [Run an agent on this branch](https://app.netlify.com/projects/superset-docs-preview/agent-runs#DctBDoQgDEDRu3StESYGmbnBHKNCURO0TcFkJsa7S_7mb94ForxLhQ9AB7PiEdb2mOmHR9SzMCoNibAOK5fay6kL9cJ5CxuVRiJJ5v83NuRwTu_X5DzaljHGO-u9HeF-AA) |
---
<!-- [superset-docs-preview Preview](https://deploy-preview-44892--superset-docs-preview.netlify.app) -->
_To edit notification comments on pull requests, go to your [Netlify project configuration](https://app.netlify.com/projects/superset-docs-preview/configuration/notifications#deploy-notifications)._
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(retention): let a host install purge policies for its own soft-delete roots [superset]
Posted by "sadpandajoe (via GitHub)" <gi...@apache.org>.
sadpandajoe commented on code in PR #44892:
URL: https://github.com/apache/superset/pull/44892#discussion_r4183962399
##########
superset/commands/deletion_retention/purge_policy.py:
##########
@@ -860,16 +894,131 @@ def declare(
cleanup_permission=cleanup_dataset_permission,
),
}
- return validate_unique_root_policies(registry.values())
+ return tuple(registry.values())
+
+
+def _host_purge_policies(
+ provider: Callable[[], Any] | None,
+) -> tuple[PurgeEntityPolicy, ...]:
+ """Return the purge policies a host installed for its own roots.
+
+ A host distribution can carry ``SoftDeleteMixin`` entities this package
+ cannot import. The retention task discovers those entities through the
+ mixin registry, so without a policy they reach the cascade as an
+ unsupported model. The host therefore declares their purge behavior and
+ installs it under ``PURGE_POLICIES_FUNC``.
+
+ Host boundary: everything about a host policy is settled here, where it is
+ admitted -- an unavailable provider, a malformed payload, a collision with
+ a root declared in this package, two declarations for one root, and a
+ declaration the shared cleanup could not execute. Each is logged and
+ dropped, so that root is reported as unsupported instead of failing row by
+ row in a scheduled run. A broken host declaration must not take the purge
+ down with it, and must never redefine how a chart, dashboard or dataset is
+ purged.
+ """
+ if provider is None:
+ return ()
+ try:
+ provided: Any = provider()
+ except Exception: # pylint: disable=broad-except
+ logger.exception(
+ "purge_policy: %s is unavailable; keeping built-in roots only",
+ HOST_POLICIES_CONFIG_KEY,
+ )
+ return ()
+ if not isinstance(provided, (list, tuple)) or not all(
+ isinstance(policy, PurgeEntityPolicy) for policy in provided
+ ):
+ logger.error(
+ "purge_policy: %s returned %s; expected a sequence of PurgeEntityPolicy",
+ HOST_POLICIES_CONFIG_KEY,
+ type(provided).__name__,
+ )
+ return ()
+ builtin_roots: frozenset[type[Any]] = frozenset(
+ policy.model for policy in _builtin_purge_policies()
+ )
+ declared: dict[type[Any], int] = {}
+ for policy in provided:
+ declared[policy.model] = declared.get(policy.model, 0) + 1
+ # Which of two declarations for one root is authoritative is undecidable,
+ # and the loser would still delete rows. Dropping both leaves the model
+ # reported as unsupported, which is the recoverable outcome. Resolving it
+ # here also keeps the duplicate away from validate_unique_root_policies,
+ # whose ValueError would abort the whole scheduled run.
+ duplicated: list[type[Any]] = [
+ model for model, count in declared.items() if count > 1
+ ]
+ for model in duplicated:
+ logger.error(
+ "purge_policy: %s declares %d policies for %s; ignoring all of them",
+ HOST_POLICIES_CONFIG_KEY,
+ declared[model],
+ model.__name__,
+ )
+ accepted: list[PurgeEntityPolicy] = []
+ for policy in provided:
+ if policy.model in builtin_roots:
+ logger.error(
+ "purge_policy: host policy for built-in root %s ignored",
+ policy.model.__name__,
+ )
+ continue
+ if policy.model in duplicated:
+ continue
+ try:
+ _validated_policy(policy)
+ except Exception as ex: # pylint: disable=broad-except
+ # Validated at admission rather than when a row is purged: a
+ # declaration the cleanup cannot execute would otherwise fail once
+ # per eligible row, counted as a cascade failure, and stay
+ # invisible until something aged past the window.
+ logger.error(
+ "purge_policy: host policy for %s rejected: %s",
+ policy.model.__name__,
+ ex,
+ )
+ continue
+ accepted.append(policy)
+ return tuple(accepted)
+
+
+def purge_policy_registry() -> Mapping[type[Any], PurgeEntityPolicy]:
+ """Index the built-in purge roots plus any the host installed.
+
+ The index is cached per installed provider, not once per process. Freezing
+ it at first use would pin whatever happened to be installed at that
+ moment -- including nothing at all, for a call made before startup
+ finished -- until a restart. Keying the cache on the provider means it is
+ invoked once however many roots are purged, a host that installs or
+ replaces one is honored, and the ordinary case (no provider) resolves to a
+ single cached index.
+ """
+ provider: Callable[[], Any] | None = (
+ current_app.config.get(HOST_POLICIES_CONFIG_KEY) if has_app_context() else None
+ )
+ return _registry_for(provider)
Review Comment:
A callable provider implemented as a mutable dataclass raises `TypeError: unhashable type` at this cache before it can be invoked or isolated, so even the built-in roots stop purging. Could this cache by provider identity without requiring extension callables to be hashable?
##########
superset/tasks/deletion_retention.py:
##########
@@ -132,6 +133,54 @@ def _report_model_counts(outcome: str, counts: dict[str, int]) -> None:
)
+@dataclass
+class _PassTotals:
+ """What one pass over the soft-delete roots produced."""
+
+ purged: dict[str, int] = field(default_factory=dict)
+ would_purge: dict[str, int] = field(default_factory=dict)
+ unsupported: dict[str, int] = field(default_factory=dict)
+ cascade_failures: int = 0
+ blocked: int = 0
+ scan_failures: int = 0
+
+
+def _purge_roots(cutoff: datetime, dry_run: bool) -> _PassTotals:
+ """Process each registered root, isolating one root's failure from the rest."""
+ totals = _PassTotals()
+ for model in _soft_delete_models():
+ entity_type = _model_table_name(model)
+ try:
+ if model not in purge_policy_registry():
+ totals.unsupported[entity_type] = 1
+ logger.warning(
+ "deletion_retention: skipping %s: no purge policy", entity_type
+ )
+ stats_logger_manager.instance.incr(
+ f"{_METRIC_PREFIX}.unsupported_models.{entity_type}"
+ )
+ continue
+ purged_n, would_n, failed_n, blocked_n = _purge_model(
+ model, cutoff, dry_run
+ )
+ except Exception: # pylint: disable=broad-except
+ # One root must not cost the others their run. The eligible-id scan
+ # runs outside _purge_model's per-entity handler, so a column it
+ # cannot read or a transient database error would otherwise abort
+ # the whole pass -- including the roots that would have purged.
+ db.session.rollback() # pylint: disable=consider-using-transaction
Review Comment:
If the first scan page purges 500 rows and the next SELECT fails, those deletions are already committed, but this handler discards the root’s accumulated counts; the summary and purge gauges report only later roots. Could scan isolation preserve the counts accumulated before the failing page?
##########
superset/commands/deletion_retention/purge_policy.py:
##########
@@ -905,32 +1056,226 @@ def _validated_purge_policy(model: type[Any]) -> PurgeEntityPolicy:
details = (
f"{details}; " if details else ""
) + f"stale_listeners=[{', '.join(coverage.stale_listeners)}]"
- raise RuntimeError(f"Incomplete purge policy for {model.__name__}: {details}")
+ raise RuntimeError(
+ f"Incomplete purge policy for {policy.model.__name__}: {details}"
+ )
return policy
+def _ownership_dependency(
+ policy: PurgeEntityPolicy, related_table: str
+) -> DependencyPolicy | None:
+ """The single owned/association edge attaching *related_table*, if clear."""
+ candidates: tuple[DependencyPolicy, ...] = tuple(
+ dependency
+ for dependency in policy.dependencies
+ if dependency.classification
+ in {DependencyClassification.OWNED, DependencyClassification.ASSOCIATION}
+ and dependency.key.related_table == related_table
+ and dependency.key.direction == "inbound"
+ )
+ return candidates[0] if len(candidates) == 1 else None
+
+
+def _validate_owned_traversal(policy: PurgeEntityPolicy) -> None:
+ """Reject an owned table reachable only through an association.
+
+ ``cascade_hard_delete`` empties associations before owned children, while
+ an owned table's predicate selects its rows *through* its ownership path
+ (see ``_owner_value_select``). If a hop on that path is an association,
+ its rows are already gone when the owned delete runs: the statement
+ matches nothing, leaving the descendants orphaned where foreign keys are
+ unenforced and blocking the root's delete where they are not.
+
+ Refused at declaration time rather than executed. The shape has a
+ remedy -- classify the intermediate table as owned, which places it in
+ the same phase as what it leads to.
+ """
+ root_table: str = sa.inspect(policy.model).local_table.name
+ for dependency in policy.dependencies:
+ if dependency.classification is not DependencyClassification.OWNED:
+ continue
+ table_name: str = dependency.key.owner_table
+ visited: set[str] = set()
+ while table_name != root_table and table_name not in visited:
+ visited.add(table_name)
+ hop: DependencyPolicy | None = _ownership_dependency(policy, table_name)
+ if hop is None:
+ # An absent or ambiguous path is reported by coverage and by
+ # _ownership_edge at execution; not this check's business.
+ break
+ if hop.classification is DependencyClassification.ASSOCIATION:
+ raise RuntimeError(
+ f"Owned dependency {dependency.key.describe()} is reachable "
+ f"only through association {hop.key.describe()}; "
+ "associations are deleted first, so the owned rows would "
+ "be orphaned"
+ )
+ table_name = hop.key.owner_table
+
+
+def _tables_share_a_foreign_key(metadata: sa.MetaData, first: str, second: str) -> bool:
+ """Whether either table declares a foreign key into the other."""
+ for owner_name, other_name in ((first, second), (second, first)):
+ table: sa.Table | None = metadata.tables.get(owner_name)
+ if table is None:
+ continue
+ for constraint in table.foreign_key_constraints:
+ if constraint.elements[0].column.table.name == other_name:
+ return True
+ return False
+
+
+def _validate_scanner_requirements(policy: PurgeEntityPolicy) -> None:
+ """Reject a root the scheduled scan cannot page through.
+
+ The retention task selects, windows and orders eligible rows by ``id``,
+ and the cascade pins each row by it. A root keyed on something else -- a
+ UUID primary key with no ``id`` column -- raises inside the scan, outside
+ the per-row error handling, so the run aborts before the remaining roots
+ are reached.
+ """
+ table: sa.Table = sa.inspect(policy.model).local_table
+ if "id" not in table.c:
+ raise RuntimeError(
+ f"Purge root {policy.model.__name__} has no 'id' column; the "
+ "scheduled scan pages eligible rows by id"
+ )
+ if "deleted_at" not in table.c:
+ raise RuntimeError(
+ f"Purge root {policy.model.__name__} has no 'deleted_at' column; "
+ "every purge path requires the row to be archived first"
+ )
+
+
+def _validate_recursive_ownership(policy: PurgeEntityPolicy) -> None:
+ """Reject a self-referencing owned table under the stock cleanup.
+
+ ``delete_owned_children`` issues one statement per declared edge, so a
+ table that owns itself is pruned one level deep: a three-level tree
+ leaves its deepest rows behind -- orphaned where foreign keys are
+ unenforced, and blocking the root's own delete where they are not. A host
+ declaring this shape supplies cleanup that walks the whole sub-tree.
+ """
+ for dependency in policy.dependencies:
+ if dependency.classification is not DependencyClassification.OWNED:
+ continue
+ key: DependencyKey = dependency.key
+ if key.owner_table != key.related_table:
+ continue
+ if policy.delete_owned_children is delete_owned_children:
+ raise RuntimeError(
+ f"Owned dependency {key.describe()} is self-referencing; the "
+ "stock owned-child cleanup deletes one level, so this policy "
+ "must supply its own delete_owned_children"
+ )
+
+
+def _validate_sibling_ownership(policy: PurgeEntityPolicy) -> None:
+ """Reject owned siblings that reference one another.
+
+ Owned deletes are sorted by ownership depth with a stable sort, so two
+ owned tables at the same depth are deleted in the order they were
+ declared. With a foreign key between them, that order decides the
+ outcome: removing the referenced table first is refused by an enforced
+ constraint and rolls the purge back. Ordering deletes by their own
+ foreign keys would be the richer fix; refusing the shape keeps the
+ engine's contract honest until something needs it.
+ """
+ metadata: sa.MetaData = sa.inspect(policy.model).local_table.metadata
+ by_depth: dict[int, list[DependencyPolicy]] = {}
+ for dependency in policy.dependencies:
+ if dependency.classification is not DependencyClassification.OWNED:
+ continue
+ depth: int = _dependency_owner_depth(policy, dependency.key)
Review Comment:
For `workflow → OWNED node` plus `node → OWNED node`, a custom recursive cleanup passes the self-reference guard, but this stock-depth calculation then raises `Expected one ownership path to node, found 2` and drops the policy as unsupported. Could custom-cleaned child trees avoid this stock traversal requirement, as self-referencing roots already do?
##########
superset/commands/deletion_retention/purge_policy.py:
##########
@@ -905,32 +1056,226 @@ def _validated_purge_policy(model: type[Any]) -> PurgeEntityPolicy:
details = (
f"{details}; " if details else ""
) + f"stale_listeners=[{', '.join(coverage.stale_listeners)}]"
- raise RuntimeError(f"Incomplete purge policy for {model.__name__}: {details}")
+ raise RuntimeError(
+ f"Incomplete purge policy for {policy.model.__name__}: {details}"
+ )
return policy
+def _ownership_dependency(
+ policy: PurgeEntityPolicy, related_table: str
+) -> DependencyPolicy | None:
+ """The single owned/association edge attaching *related_table*, if clear."""
+ candidates: tuple[DependencyPolicy, ...] = tuple(
+ dependency
+ for dependency in policy.dependencies
+ if dependency.classification
+ in {DependencyClassification.OWNED, DependencyClassification.ASSOCIATION}
+ and dependency.key.related_table == related_table
+ and dependency.key.direction == "inbound"
+ )
+ return candidates[0] if len(candidates) == 1 else None
+
+
+def _validate_owned_traversal(policy: PurgeEntityPolicy) -> None:
+ """Reject an owned table reachable only through an association.
+
+ ``cascade_hard_delete`` empties associations before owned children, while
+ an owned table's predicate selects its rows *through* its ownership path
+ (see ``_owner_value_select``). If a hop on that path is an association,
+ its rows are already gone when the owned delete runs: the statement
+ matches nothing, leaving the descendants orphaned where foreign keys are
+ unenforced and blocking the root's delete where they are not.
+
+ Refused at declaration time rather than executed. The shape has a
+ remedy -- classify the intermediate table as owned, which places it in
+ the same phase as what it leads to.
+ """
+ root_table: str = sa.inspect(policy.model).local_table.name
+ for dependency in policy.dependencies:
+ if dependency.classification is not DependencyClassification.OWNED:
+ continue
+ table_name: str = dependency.key.owner_table
+ visited: set[str] = set()
+ while table_name != root_table and table_name not in visited:
+ visited.add(table_name)
+ hop: DependencyPolicy | None = _ownership_dependency(policy, table_name)
+ if hop is None:
+ # An absent or ambiguous path is reported by coverage and by
+ # _ownership_edge at execution; not this check's business.
+ break
+ if hop.classification is DependencyClassification.ASSOCIATION:
+ raise RuntimeError(
+ f"Owned dependency {dependency.key.describe()} is reachable "
+ f"only through association {hop.key.describe()}; "
+ "associations are deleted first, so the owned rows would "
+ "be orphaned"
+ )
+ table_name = hop.key.owner_table
+
+
+def _tables_share_a_foreign_key(metadata: sa.MetaData, first: str, second: str) -> bool:
+ """Whether either table declares a foreign key into the other."""
+ for owner_name, other_name in ((first, second), (second, first)):
+ table: sa.Table | None = metadata.tables.get(owner_name)
+ if table is None:
+ continue
+ for constraint in table.foreign_key_constraints:
+ if constraint.elements[0].column.table.name == other_name:
+ return True
+ return False
+
+
+def _validate_scanner_requirements(policy: PurgeEntityPolicy) -> None:
+ """Reject a root the scheduled scan cannot page through.
+
+ The retention task selects, windows and orders eligible rows by ``id``,
+ and the cascade pins each row by it. A root keyed on something else -- a
+ UUID primary key with no ``id`` column -- raises inside the scan, outside
+ the per-row error handling, so the run aborts before the remaining roots
+ are reached.
+ """
+ table: sa.Table = sa.inspect(policy.model).local_table
+ if "id" not in table.c:
+ raise RuntimeError(
+ f"Purge root {policy.model.__name__} has no 'id' column; the "
+ "scheduled scan pages eligible rows by id"
+ )
+ if "deleted_at" not in table.c:
+ raise RuntimeError(
+ f"Purge root {policy.model.__name__} has no 'deleted_at' column; "
+ "every purge path requires the row to be archived first"
+ )
+
+
+def _validate_recursive_ownership(policy: PurgeEntityPolicy) -> None:
+ """Reject a self-referencing owned table under the stock cleanup.
+
+ ``delete_owned_children`` issues one statement per declared edge, so a
+ table that owns itself is pruned one level deep: a three-level tree
+ leaves its deepest rows behind -- orphaned where foreign keys are
+ unenforced, and blocking the root's own delete where they are not. A host
+ declaring this shape supplies cleanup that walks the whole sub-tree.
+ """
+ for dependency in policy.dependencies:
+ if dependency.classification is not DependencyClassification.OWNED:
+ continue
+ key: DependencyKey = dependency.key
+ if key.owner_table != key.related_table:
+ continue
+ if policy.delete_owned_children is delete_owned_children:
+ raise RuntimeError(
+ f"Owned dependency {key.describe()} is self-referencing; the "
+ "stock owned-child cleanup deletes one level, so this policy "
+ "must supply its own delete_owned_children"
+ )
+
+
+def _validate_sibling_ownership(policy: PurgeEntityPolicy) -> None:
+ """Reject owned siblings that reference one another.
+
+ Owned deletes are sorted by ownership depth with a stable sort, so two
+ owned tables at the same depth are deleted in the order they were
+ declared. With a foreign key between them, that order decides the
+ outcome: removing the referenced table first is refused by an enforced
+ constraint and rolls the purge back. Ordering deletes by their own
+ foreign keys would be the richer fix; refusing the shape keeps the
+ engine's contract honest until something needs it.
+ """
+ metadata: sa.MetaData = sa.inspect(policy.model).local_table.metadata
+ by_depth: dict[int, list[DependencyPolicy]] = {}
+ for dependency in policy.dependencies:
+ if dependency.classification is not DependencyClassification.OWNED:
+ continue
+ depth: int = _dependency_owner_depth(policy, dependency.key)
+ by_depth.setdefault(depth, []).append(dependency)
+ for siblings in by_depth.values():
+ for index, first in enumerate(siblings):
+ for second in siblings[index + 1 :]:
+ if first.key.related_table == second.key.related_table:
+ continue
+ if _tables_share_a_foreign_key(
+ metadata, first.key.related_table, second.key.related_table
+ ):
+ raise RuntimeError(
+ f"Owned dependencies {first.key.describe()} and "
+ f"{second.key.describe()} reference each other; owned "
+ "deletes run in declaration order, so whether this "
+ "graph purges depends on it"
+ )
+
+
+#: Classifications the shared cleanup executes as SQL, through
+#: ``_dependency_predicates`` -- which only knows how to read an inbound
+#: foreign key.
+_EXECUTABLE_CLASSIFICATIONS: frozenset[DependencyClassification] = frozenset(
+ {
+ DependencyClassification.OWNED,
+ DependencyClassification.ASSOCIATION,
+ DependencyClassification.BLOCK,
+ }
+)
+
+
def _validate_executable_declarations(policy: PurgeEntityPolicy) -> None:
- """Reject executable classifications missing their required action metadata."""
+ """Reject declarations the shared cleanup could not carry out.
+
+ Each shape here would otherwise raise mid-purge: for the scheduled task,
+ after the write-ahead audit row is already committed, so the attempt is
+ recorded as a failure on every run. Refused at registration instead.
+ """
+ metadata: sa.MetaData = sa.inspect(policy.model).local_table.metadata
for dependency in policy.dependencies:
+ key: DependencyKey = dependency.key
+ if dependency.classification in _EXECUTABLE_CLASSIFICATIONS:
+ if key.kind != "foreign_key" or key.direction != "inbound":
+ raise RuntimeError(
+ f"{dependency.classification.value} dependency "
+ f"{key.describe()} is not an inbound foreign key; the "
+ "shared cleanup can only act on those"
+ )
+ if key.related_table not in metadata.tables:
+ raise RuntimeError(
+ f"{dependency.classification.value} dependency "
+ f"{key.describe()} names a table outside the root's "
+ "metadata, which cleanup cannot resolve"
+ )
if (
- dependency.classification is DependencyClassification.LISTENER_EFFECT
- and dependency.listener_action is None
+ dependency.classification is DependencyClassification.BLOCK
+ and dependency.blocker is None
):
- raise RuntimeError(
- f"Missing listener action for {dependency.key.describe()}"
- )
+ raise RuntimeError(f"Missing blocker reason for {key.describe()}")
+ if dependency.classification is DependencyClassification.LISTENER_EFFECT:
+ if dependency.listener_action is None:
+ raise RuntimeError(
+ f"Missing listener action for {dependency.key.describe()}"
+ )
+ if dependency.phase is None:
Review Comment:
A graph-complete listener declaration with `DELETE_DATASET_PERMISSION` at `ExecutionPhase.OWNED` passes admission, but stock callbacks dispatch listener effects only in `ASSOCIATIONS` and `POST_DELETE`, so the root can be purged without its declared cleanup ever running. Could admission validate the action/phase combinations the selected callbacks actually execute?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(retention): let a host install purge policies for its own soft-delete roots [superset]
Posted by "alexandrusoare (via GitHub)" <gi...@apache.org>.
alexandrusoare commented on code in PR #44892:
URL: https://github.com/apache/superset/pull/44892#discussion_r4183597338
##########
superset/tasks/deletion_retention.py:
##########
@@ -132,6 +133,54 @@ def _report_model_counts(outcome: str, counts: dict[str, int]) -> None:
)
+@dataclass
+class _PassTotals:
+ """What one pass over the soft-delete roots produced."""
+
+ purged: dict[str, int] = field(default_factory=dict)
+ would_purge: dict[str, int] = field(default_factory=dict)
+ unsupported: dict[str, int] = field(default_factory=dict)
+ cascade_failures: int = 0
+ blocked: int = 0
+ scan_failures: int = 0
+
+
+def _purge_roots(cutoff: datetime, dry_run: bool) -> _PassTotals:
+ """Process each registered root, isolating one root's failure from the rest."""
+ totals = _PassTotals()
+ for model in _soft_delete_models():
+ entity_type = _model_table_name(model)
+ try:
+ if model not in purge_policy_registry():
+ totals.unsupported[entity_type] = 1
+ logger.warning(
+ "deletion_retention: skipping %s: no purge policy", entity_type
+ )
+ stats_logger_manager.instance.incr(
+ f"{_METRIC_PREFIX}.unsupported_models.{entity_type}"
+ )
+ continue
+ purged_n, would_n, failed_n, blocked_n = _purge_model(
+ model, cutoff, dry_run
+ )
+ except Exception: # pylint: disable=broad-except
Review Comment:
Policies are now validated when they're admitted, logged with the reason and dropped, so their root reports as unsupported and the task never sees one.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(retention): let a host install purge policies for its own soft-delete roots [superset]
Posted by "alexandrusoare (via GitHub)" <gi...@apache.org>.
alexandrusoare commented on code in PR #44892:
URL: https://github.com/apache/superset/pull/44892#discussion_r4183601606
##########
superset/commands/deletion_retention/purge_policy.py:
##########
@@ -909,23 +1032,211 @@ def _validated_purge_policy(model: type[Any]) -> PurgeEntityPolicy:
return policy
+def _ownership_dependency(
+ policy: PurgeEntityPolicy, related_table: str
+) -> DependencyPolicy | None:
+ """The single owned/association edge attaching *related_table*, if clear."""
+ candidates: tuple[DependencyPolicy, ...] = tuple(
+ dependency
+ for dependency in policy.dependencies
+ if dependency.classification
+ in {DependencyClassification.OWNED, DependencyClassification.ASSOCIATION}
+ and dependency.key.related_table == related_table
+ and dependency.key.direction == "inbound"
+ )
+ return candidates[0] if len(candidates) == 1 else None
+
+
+def _validate_owned_traversal(policy: PurgeEntityPolicy) -> None:
+ """Reject an owned table reachable only through an association.
+
+ ``cascade_hard_delete`` empties associations before owned children, while
+ an owned table's predicate selects its rows *through* its ownership path
+ (see ``_owner_value_select``). If a hop on that path is an association,
+ its rows are already gone when the owned delete runs: the statement
+ matches nothing, leaving the descendants orphaned where foreign keys are
+ unenforced and blocking the root's delete where they are not.
+
+ Refused at declaration time rather than executed. The shape has a
+ remedy -- classify the intermediate table as owned, which places it in
+ the same phase as what it leads to.
+ """
+ root_table: str = sa.inspect(policy.model).local_table.name
+ for dependency in policy.dependencies:
+ if dependency.classification is not DependencyClassification.OWNED:
+ continue
+ table_name: str = dependency.key.owner_table
+ visited: set[str] = set()
+ while table_name != root_table and table_name not in visited:
+ visited.add(table_name)
+ hop: DependencyPolicy | None = _ownership_dependency(policy, table_name)
+ if hop is None:
+ # An absent or ambiguous path is reported by coverage and by
+ # _ownership_edge at execution; not this check's business.
+ break
+ if hop.classification is DependencyClassification.ASSOCIATION:
+ raise RuntimeError(
+ f"Owned dependency {dependency.key.describe()} is reachable "
+ f"only through association {hop.key.describe()}; "
+ "associations are deleted first, so the owned rows would "
+ "be orphaned"
+ )
+ table_name = hop.key.owner_table
+
+
+def _tables_share_a_foreign_key(metadata: sa.MetaData, first: str, second: str) -> bool:
Review Comment:
A foreign key constraint references exactly one table, so elements[0] covers multi-column keys. Indirect references it does miss — deliberately, since two siblings pointing at a common third table have no ordering constraint between them, and checking transitively would reject shapes that work
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(retention): let a host install purge policies for its own soft-delete roots [superset]
Posted by "alexandrusoare (via GitHub)" <gi...@apache.org>.
alexandrusoare commented on code in PR #44892:
URL: https://github.com/apache/superset/pull/44892#discussion_r4165621129
##########
superset/commands/deletion_retention/purge_policy.py:
##########
@@ -860,7 +867,96 @@ def declare(
cleanup_permission=cleanup_dataset_permission,
),
}
- return validate_unique_root_policies(registry.values())
+ return tuple(registry.values())
+
+
+def _host_purge_policies() -> tuple[PurgeEntityPolicy, ...]:
+ """Return the purge policies a host installed for its own roots.
+
+ A host distribution can carry ``SoftDeleteMixin`` entities this package
+ cannot import. The retention task discovers those entities through the
+ mixin registry, so without a policy they reach the cascade as an
+ unsupported model. The host therefore declares their purge behavior and
+ installs it under ``PURGE_POLICIES_FUNC``.
+
+ Host boundary: an unavailable provider, a malformed payload, a policy that
+ collides with a root declared here, and two policies for one host root are
+ each logged and dropped. A broken host declaration must not take the
+ scheduled purge down with it, and must never redefine how a chart,
+ dashboard or dataset is purged.
+ """
+ if not has_app_context():
+ return ()
+ provider: Callable[[], Any] | None = current_app.config.get(
+ HOST_POLICIES_CONFIG_KEY
+ )
+ if provider is None:
+ return ()
+ try:
+ provided: Any = provider()
+ except Exception: # pylint: disable=broad-except
+ logger.exception(
+ "purge_policy: %s is unavailable; keeping built-in roots only",
+ HOST_POLICIES_CONFIG_KEY,
+ )
+ return ()
+ if not isinstance(provided, (list, tuple)) or not all(
+ isinstance(policy, PurgeEntityPolicy) for policy in provided
+ ):
+ logger.error(
+ "purge_policy: %s returned %s; expected a sequence of PurgeEntityPolicy",
+ HOST_POLICIES_CONFIG_KEY,
+ type(provided).__name__,
+ )
+ return ()
+ builtin_roots: frozenset[type[Any]] = frozenset(
+ policy.model for policy in _builtin_purge_policies()
+ )
+ declared: dict[type[Any], int] = {}
+ for policy in provided:
+ declared[policy.model] = declared.get(policy.model, 0) + 1
+ # Which of two declarations for one root is authoritative is undecidable,
+ # and the loser would still delete rows. Dropping both leaves the model
+ # reported as unsupported, which is the recoverable outcome. Resolving it
+ # here also keeps the duplicate away from validate_unique_root_policies,
+ # whose ValueError would abort the whole scheduled run.
+ duplicated: list[type[Any]] = [
+ model for model, count in declared.items() if count > 1
+ ]
+ for model in duplicated:
+ logger.error(
+ "purge_policy: %s declares %d policies for %s; ignoring all of them",
+ HOST_POLICIES_CONFIG_KEY,
+ declared[model],
+ model.__name__,
+ )
+ accepted: list[PurgeEntityPolicy] = []
+ for policy in provided:
+ if policy.model in builtin_roots:
+ logger.error(
+ "purge_policy: host policy for built-in root %s ignored",
+ policy.model.__name__,
+ )
+ continue
+ if policy.model in duplicated:
+ continue
+ accepted.append(policy)
+ return tuple(accepted)
+
+
+def purge_policy_registry() -> Mapping[type[Any], PurgeEntityPolicy]:
+ """Index the built-in purge roots plus any the host installed.
+
+ Uncached on purpose, unlike its two inputs: the built-in declarations are
+ built once per process, while a host policy is resolved per call, so a
+ provider installed after the first purge is honored for any root not yet
+ resolved. ``get_purge_policy`` memoizes per model, so replacing the policy
+ of a root it has already resolved needs a restart -- and that memoization
+ is why rebuilding this small index is off the hot path.
+ """
+ return validate_unique_root_policies(
Review Comment:
Good catch — associations are emptied before owned children, so an owned table reached through one selects from rows that are already gone. Went with your first option: validation now rejects that shape and points at the fix (declare the intermediate table owned).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(retention): let a host install purge policies for its own soft-delete roots [superset]
Posted by "codeant-ai-for-open-source[bot] (via GitHub)" <gi...@apache.org>.
codeant-ai-for-open-source[bot] commented on code in PR #44892:
URL: https://github.com/apache/superset/pull/44892#discussion_r4165429857
##########
superset/commands/deletion_retention/purge_policy.py:
##########
@@ -860,7 +867,74 @@ def declare(
cleanup_permission=cleanup_dataset_permission,
),
}
- return validate_unique_root_policies(registry.values())
+ return tuple(registry.values())
+
+
+def _host_purge_policies() -> tuple[PurgeEntityPolicy, ...]:
+ """Return the purge policies a host installed for its own roots.
+
+ A host distribution can carry ``SoftDeleteMixin`` entities this package
+ cannot import. The retention task discovers those entities through the
+ mixin registry, so without a policy they reach the cascade as an
+ unsupported model. The host therefore declares their purge behavior and
+ installs it under ``PURGE_POLICIES_FUNC``.
+
+ Host boundary: an unavailable provider, a malformed payload, or a policy
+ that collides with a root declared here is logged and dropped. A broken
+ host declaration must not take the scheduled purge down with it, and must
+ never redefine how a chart, dashboard or dataset is purged.
+ """
+ if not has_app_context():
+ return ()
+ provider: Callable[[], Any] | None = current_app.config.get(
+ HOST_POLICIES_CONFIG_KEY
+ )
+ if provider is None:
+ return ()
+ try:
+ provided: Any = provider()
+ except Exception: # pylint: disable=broad-except
+ logger.exception(
+ "purge_policy: %s is unavailable; keeping built-in roots only",
+ HOST_POLICIES_CONFIG_KEY,
+ )
+ return ()
+ if not isinstance(provided, (list, tuple)) or not all(
+ isinstance(policy, PurgeEntityPolicy) for policy in provided
+ ):
+ logger.error(
+ "purge_policy: %s returned %s; expected a sequence of PurgeEntityPolicy",
+ HOST_POLICIES_CONFIG_KEY,
+ type(provided).__name__,
+ )
+ return ()
+ builtin_roots: frozenset[type[Any]] = frozenset(
+ policy.model for policy in _builtin_purge_policies()
+ )
+ accepted: list[PurgeEntityPolicy] = []
+ for policy in provided:
+ if policy.model in builtin_roots:
+ logger.error(
+ "purge_policy: host policy for built-in root %s ignored",
+ policy.model.__name__,
+ )
+ continue
+ accepted.append(policy)
+ return tuple(accepted)
Review Comment:
✅ **CodeAnt verified this suggestion was addressed in subsequent commits and marked this thread resolved** as of `d2715f1`.
Duplicate non-built-in models are counted and all policies for duplicated models are excluded from the accepted host policies before registry validation, preventing `validate_unique_root_policies` from aborting the purge.
<sub>If that's not right, unresolve this thread and CodeAnt will leave it open.</sub>
<!-- codeant-auto-resolve-reply -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(retention): let a host install purge policies for its own soft-delete roots [superset]
Posted by "sadpandajoe (via GitHub)" <gi...@apache.org>.
sadpandajoe commented on code in PR #44892:
URL: https://github.com/apache/superset/pull/44892#discussion_r4206852292
##########
superset/commands/deletion_retention/purge_policy.py:
##########
@@ -905,32 +1414,308 @@ def _validated_purge_policy(model: type[Any]) -> PurgeEntityPolicy:
details = (
f"{details}; " if details else ""
) + f"stale_listeners=[{', '.join(coverage.stale_listeners)}]"
- raise RuntimeError(f"Incomplete purge policy for {model.__name__}: {details}")
+ raise RuntimeError(
+ f"Incomplete purge policy for {policy.model.__name__}: {details}"
+ )
return policy
-def _validate_executable_declarations(policy: PurgeEntityPolicy) -> None:
- """Reject executable classifications missing their required action metadata."""
+def _ownership_dependency(
+ policy: PurgeEntityPolicy, related_table: str
+) -> DependencyPolicy | None:
+ """The single owned/association edge attaching *related_table*, if clear."""
+ candidates: tuple[DependencyPolicy, ...] = tuple(
+ dependency
+ for dependency in policy.dependencies
+ if dependency.classification
+ in {DependencyClassification.OWNED, DependencyClassification.ASSOCIATION}
+ and dependency.key.related_table == related_table
+ and dependency.key.direction == "inbound"
+ )
+ return candidates[0] if len(candidates) == 1 else None
+
+
+def _validate_owned_traversal(policy: PurgeEntityPolicy) -> None:
+ """Reject an owned table reachable only through an association.
+
+ The shared cleanup empties associations before owned children, while an
+ owned table's predicate selects its rows *through* its ownership path. If
+ a hop on that path is an association, its rows are already gone when the
+ owned delete runs, so the statement matches nothing.
+
+ Kept where the rest of the ordering rules were dropped because this one
+ can be silent: where foreign keys are enforced the association delete
+ fails loudly, but where they are not -- SQLite -- the root is purged and
+ its descendants are left orphaned with nothing reported. The remedy is to
+ classify the intermediate table as owned, which places it in the same
+ phase as what it leads to.
+ """
+ root_table: str = sa.inspect(policy.model).local_table.name
for dependency in policy.dependencies:
- if (
- dependency.classification is DependencyClassification.LISTENER_EFFECT
- and dependency.listener_action is None
- ):
- raise RuntimeError(
- f"Missing listener action for {dependency.key.describe()}"
- )
- if (
- dependency.classification is DependencyClassification.VERSION_OWNED
- and dependency.version_column is None
+ if dependency.classification is not DependencyClassification.OWNED:
+ continue
+ table_name: str = dependency.key.owner_table
+ visited: set[str] = set()
+ while table_name != root_table and table_name not in visited:
+ visited.add(table_name)
+ hop: DependencyPolicy | None = _ownership_dependency(policy, table_name)
+ if hop is None:
+ # An absent or ambiguous path is reported by coverage, and by
+ # _ownership_edge at execution; not this check's business.
+ break
+ if hop.classification is DependencyClassification.ASSOCIATION:
+ raise RuntimeError(
+ f"Owned dependency {dependency.key.describe()} is reachable "
+ f"only through association {hop.key.describe()}; "
+ "associations are deleted first, so the owned rows would "
+ "be orphaned"
+ )
+ table_name = hop.key.owner_table
+
+
+def _validate_recursive_ownership(policy: PurgeEntityPolicy) -> None:
+ """Reject a self-referencing owned table under one-level cleanup.
+
+ Cleanup that issues one statement per declared edge prunes a table that
+ owns itself one level deep. Where foreign keys are enforced the root's own
+ delete then fails and rolls back; where they are not -- SQLite -- the root
+ is purged and its grandchildren are left behind with a dangling parent id
+ and nothing reported. A policy declaring a tree walks it itself and says
+ so with ``walks_own_subtree``.
+ """
+ for dependency in policy.dependencies:
+ if dependency.classification is not DependencyClassification.OWNED:
Review Comment:
A host root with a `parent_id` self-FK that classifies the discovered edge as `ASSOCIATION` skips this guard (only `OWNED` is refused here), so the stock association cleanup runs `DELETE FROM host_node WHERE parent_id IN (SELECT id FROM host_node WHERE id = :root)`. That hard-deletes the live, non-archived direct children of the purged row with no eligibility predicate or audit row, and leaves grandchildren with a dangling `parent_id` (on SQLite, or when the child is a leaf, the purge commits as a clean success). The same applies to an `ASSOCIATION`/`OWNED` edge whose `related_table` is another purge root's table. Should admission refuse executable declarations whose related table is the root's own table or another registered root's table, regardless of `walks_own_subtree`?
##########
superset/commands/deletion_retention/purge_policy.py:
##########
@@ -860,16 +931,443 @@ def declare(
cleanup_permission=cleanup_dataset_permission,
),
}
- return validate_unique_root_policies(registry.values())
+ return tuple(registry.values())
+
+
+def _policy_label(policy: Any) -> str:
+ """A log-safe name for a payload whose shape is not yet established."""
+ model: Any = getattr(policy, "model", None)
+ return str(getattr(model, "__name__", type(model).__name__))
+
+
+def _host_policy_payload(provided: Any) -> tuple[Any, ...]:
+ """Materialize the provider's payload, or reject its shape.
+
+ Any iterable is accepted -- a list, a tuple, ``dict.values()``, a
+ generator -- since the documented contract is a sequence of policies, not
+ one particular container. Strings and bytes are excluded because they
+ iterate into characters, which would read as a sequence of bad members
+ rather than the wrong type. Materialized once: a generator cannot be
+ walked twice.
+
+ An exception raised *while reading* a lazy payload is left to propagate.
+ Walking a generator runs host code just as calling the provider does, so
+ the two are the same kind of failure and earn the same retry; only the
+ shape of a payload that arrived intact is settled for good.
+ """
+ if isinstance(provided, (str, bytes)) or not isinstance(provided, Iterable):
+ logger.error(
+ "purge_policy: %s returned %s; expected an iterable of PurgeEntityPolicy",
+ HOST_POLICIES_CONFIG_KEY,
+ type(provided).__name__,
+ )
+ return ()
+ return tuple(provided)
-@lru_cache(maxsize=None)
-def _validated_purge_policy(model: type[Any]) -> PurgeEntityPolicy:
- """Validate and return one root policy without blocking unrelated roots."""
+def _unqualified_table_conflict(policy: PurgeEntityPolicy) -> str | None:
+ """Return the first table a bare-name lookup would resolve wrongly.
+
+ Discovery records table identities by bare name while a ``MetaData`` keys
+ them by schema, so a declared ``child`` resolves to the default-schema
+ table even where the policy meant ``host.child`` -- and the cleanup would
+ delete rows belonging to unrelated roots. Carrying schema-qualified
+ identities through discovery and execution is the better fix; until then
+ the ambiguous shape is refused rather than silently mis-resolved.
+ """
+ root_table: sa.Table = sa.inspect(policy.model).local_table
+ metadata: sa.MetaData = root_table.metadata
+ keys_by_name: dict[str, list[str]] = {}
+ for table in metadata.tables.values():
+ keys_by_name.setdefault(table.name, []).append(table.key)
+ declared: set[str] = {root_table.name} | {
+ dependency.key.related_table
+ for dependency in policy.dependencies
+ # Version shadows are resolved by bare name as well, in
+ # _entity_version_targets.
+ if dependency.classification in _EXECUTABLE_CLASSIFICATIONS
+ or dependency.classification is DependencyClassification.VERSION_OWNED
+ }
+ for name in sorted(declared):
+ keys: list[str] = keys_by_name.get(name, [])
+ if len(keys) != 1 or keys[0] != name:
+ return name
+ return None
+
+
+def _admitted_host_policy(
+ candidate: Any,
+ builtin_roots: frozenset[type[Any]],
+ builtin_tables: frozenset[str],
+ builtin_types: frozenset[str],
+) -> PurgeEntityPolicy | None:
+ """Return *candidate* if it is a usable host policy, else ``None``.
+
+ Every check runs behind one handler rather than guarding each attribute:
+ the payload is host-supplied, so reading ``model`` or hashing it may
+ itself raise -- and an exception escaping here would break
+ ``purge_policy_registry`` for the built-in roots too, which is the
+ opposite of what this boundary is for.
+ """
try:
- policy: PurgeEntityPolicy = purge_policy_registry()[model]
- except KeyError as ex:
- raise ValueError(f"Unsupported purge model: {model.__name__}") from ex
+ if not isinstance(candidate, PurgeEntityPolicy):
+ logger.error(
+ "purge_policy: %s returned a %s; expected PurgeEntityPolicy",
+ HOST_POLICIES_CONFIG_KEY,
+ type(candidate).__name__,
+ )
+ return None
+ if not isinstance(candidate.model, type):
+ # Everything downstream treats the root as a mapped class:
+ # sa.inspect, the registry index, the duplicate count.
+ logger.error(
+ "purge_policy: host policy root %s is not a class",
+ _policy_label(candidate),
+ )
+ return None
+ if candidate.model in builtin_roots:
+ logger.error(
+ "purge_policy: host policy for built-in root %s ignored",
+ candidate.model.__name__,
+ )
+ return None
+ root_table: sa.Table = sa.inspect(candidate.model).local_table
+ if root_table.key in builtin_tables:
+ # A different class mapped onto a built-in table -- a subclass of
+ # Slice, say -- is not the built-in root by identity, but purging
+ # it runs host cleanup against this package's own rows. It also
+ # joins the soft-delete registry through the mixin, so the
+ # scheduled task would scan it.
+ logger.error(
+ "purge_policy: host policy root %s is mapped onto built-in "
+ "table %r; ignored",
+ candidate.model.__name__,
+ root_table.key,
+ )
+ return None
+ if not isinstance(candidate.entity_type, str) or not candidate.entity_type:
+ # Read as a string by the cascade, the audit record and every log
+ # line that names the root.
+ logger.error(
+ "purge_policy: host policy for %s declares entity_type %r",
+ candidate.model.__name__,
+ candidate.entity_type,
+ )
+ return None
+ if candidate.entity_type in builtin_types:
Review Comment:
Admission reserves built-in models, tables and `entity_type` values, but the version cascade still keys `version_changes` cleanup on the bare class name (`purge_cascade.py` maps `model.__name__` through `ENTITY_KIND_BY_CLASS_NAME`). A versioned host class that happens to be named `Dashboard` (own table, own `entity_type`) passes admission, and purging its row 7 then deletes the `version_changes` rows for the built-in dashboard 7, including a live one. Should admission also reject a host model whose `__name__` is in `ENTITY_KIND_BY_CLASS_NAME`, or should the cleanup bind to the policy's model identity instead?
##########
superset/commands/deletion_retention/purge_policy.py:
##########
@@ -905,32 +1414,308 @@ def _validated_purge_policy(model: type[Any]) -> PurgeEntityPolicy:
details = (
f"{details}; " if details else ""
) + f"stale_listeners=[{', '.join(coverage.stale_listeners)}]"
- raise RuntimeError(f"Incomplete purge policy for {model.__name__}: {details}")
+ raise RuntimeError(
+ f"Incomplete purge policy for {policy.model.__name__}: {details}"
+ )
return policy
-def _validate_executable_declarations(policy: PurgeEntityPolicy) -> None:
- """Reject executable classifications missing their required action metadata."""
+def _ownership_dependency(
+ policy: PurgeEntityPolicy, related_table: str
+) -> DependencyPolicy | None:
+ """The single owned/association edge attaching *related_table*, if clear."""
+ candidates: tuple[DependencyPolicy, ...] = tuple(
+ dependency
+ for dependency in policy.dependencies
+ if dependency.classification
+ in {DependencyClassification.OWNED, DependencyClassification.ASSOCIATION}
+ and dependency.key.related_table == related_table
+ and dependency.key.direction == "inbound"
+ )
+ return candidates[0] if len(candidates) == 1 else None
+
+
+def _validate_owned_traversal(policy: PurgeEntityPolicy) -> None:
+ """Reject an owned table reachable only through an association.
+
+ The shared cleanup empties associations before owned children, while an
+ owned table's predicate selects its rows *through* its ownership path. If
+ a hop on that path is an association, its rows are already gone when the
+ owned delete runs, so the statement matches nothing.
+
+ Kept where the rest of the ordering rules were dropped because this one
+ can be silent: where foreign keys are enforced the association delete
+ fails loudly, but where they are not -- SQLite -- the root is purged and
+ its descendants are left orphaned with nothing reported. The remedy is to
+ classify the intermediate table as owned, which places it in the same
+ phase as what it leads to.
+ """
+ root_table: str = sa.inspect(policy.model).local_table.name
for dependency in policy.dependencies:
- if (
- dependency.classification is DependencyClassification.LISTENER_EFFECT
- and dependency.listener_action is None
- ):
- raise RuntimeError(
- f"Missing listener action for {dependency.key.describe()}"
- )
- if (
- dependency.classification is DependencyClassification.VERSION_OWNED
- and dependency.version_column is None
+ if dependency.classification is not DependencyClassification.OWNED:
+ continue
+ table_name: str = dependency.key.owner_table
+ visited: set[str] = set()
+ while table_name != root_table and table_name not in visited:
+ visited.add(table_name)
+ hop: DependencyPolicy | None = _ownership_dependency(policy, table_name)
+ if hop is None:
+ # An absent or ambiguous path is reported by coverage, and by
+ # _ownership_edge at execution; not this check's business.
+ break
+ if hop.classification is DependencyClassification.ASSOCIATION:
+ raise RuntimeError(
+ f"Owned dependency {dependency.key.describe()} is reachable "
+ f"only through association {hop.key.describe()}; "
+ "associations are deleted first, so the owned rows would "
+ "be orphaned"
+ )
+ table_name = hop.key.owner_table
+
+
+def _validate_recursive_ownership(policy: PurgeEntityPolicy) -> None:
+ """Reject a self-referencing owned table under one-level cleanup.
+
+ Cleanup that issues one statement per declared edge prunes a table that
+ owns itself one level deep. Where foreign keys are enforced the root's own
+ delete then fails and rolls back; where they are not -- SQLite -- the root
+ is purged and its grandchildren are left behind with a dangling parent id
+ and nothing reported. A policy declaring a tree walks it itself and says
+ so with ``walks_own_subtree``.
+ """
+ for dependency in policy.dependencies:
+ if dependency.classification is not DependencyClassification.OWNED:
+ continue
+ key: DependencyKey = dependency.key
+ if key.owner_table != key.related_table:
+ continue
+ raise RuntimeError(
+ f"Owned dependency {key.describe()} is self-referencing; the stock "
+ "owned-child cleanup deletes one level, so this policy must supply "
+ "its own delete_owned_children"
+ )
+
+
+def _validate_scanner_requirements(policy: PurgeEntityPolicy) -> None:
+ """Reject a root the scheduled scan cannot page through.
+
+ The retention task selects, windows and orders eligible rows by ``id``,
+ and the cascade pins each row by it. A root keyed on something else -- a
+ UUID primary key with no ``id`` column -- raises inside the scan, outside
+ the per-row error handling, so the run aborts before the remaining roots
+ are reached.
+ """
+ mapper: Mapper[Any] = sa.inspect(policy.model)
+ table: sa.Table = mapper.local_table
+ if "id" not in table.c:
+ raise RuntimeError(
+ f"Purge root {policy.model.__name__} has no 'id' column; the "
+ "scheduled scan pages eligible rows by id"
+ )
+ primary_key: tuple[sa.Column[Any], ...] = tuple(mapper.primary_key)
+ if (
+ len(primary_key) != 1
+ or primary_key[0].name != "id"
+ or not isinstance(primary_key[0].type, sa.Integer)
+ ):
+ # The scan pages by id and the row is then fetched with a scalar
+ # Session.get, so a composite or differently named key matches
+ # nothing: every row fails while a dry run still counts it eligible.
+ raise RuntimeError(
+ f"Purge root {policy.model.__name__} is keyed on "
+ f"({', '.join(column.name for column in primary_key)}); the "
+ "scheduled purge pages an integer 'id' watermark and looks rows "
+ "up by it"
+ )
+ if "deleted_at" not in table.c:
+ raise RuntimeError(
+ f"Purge root {policy.model.__name__} has no 'deleted_at' column; "
+ "every purge path requires the row to be archived first"
+ )
+ uuid_column: sa.Column[Any] | None = table.c.get("uuid")
+ if (
+ uuid_column is not None
+ and isinstance(uuid_column.type, sa.Uuid)
+ and not uuid_column.type.native_uuid
+ ):
+ # The third column the frame reads: the locked claim and the
+ # conditional delete both carry ``uuid == str(entity.uuid)``. A
+ # character-based UUID type binds through a processor that expects a
+ # UUID object, so every eligible row would raise and stay archived
+ # while a dry run counted it purgeable -- on every dialect, which is
+ # what makes it checkable here.
+ #
+ # ``native_uuid`` is left alone deliberately: it has no bind processor
+ # where the database has a UUID type of its own, so the string binds
+ # cleanly on Postgres and only a database without one rejects it. That
+ # is a per-row failure like any other, and the docs say so.
+ # ``sqlalchemy_utils.UUIDType`` is not a subclass of this at all and
+ # accepts the string everywhere, which is why the built-in roots are
+ # unaffected.
+ raise RuntimeError(
+ f"Purge root {policy.model.__name__} types 'uuid' as a "
+ "character-based sa.Uuid; the locked claim binds that value as a "
+ "string, which the type rejects on every dialect"
+ )
+
+
+#: Classifications the shared cleanup executes as SQL, through
+#: ``_dependency_predicates`` -- which only knows how to read an inbound
+#: foreign key.
+_EXECUTABLE_CLASSIFICATIONS: frozenset[DependencyClassification] = frozenset(
+ {
+ DependencyClassification.OWNED,
+ DependencyClassification.ASSOCIATION,
+ DependencyClassification.BLOCK,
+ }
+)
+
+
+def _validate_versioned_root(policy: PurgeEntityPolicy) -> None:
+ """Reject version targets on a root that has no version class.
+
+ ``_delete_version_history`` resolves the *root's* version class first and
+ returns as soon as that raises, so a target declared on a child's shadow
+ is never reached for an unversioned root: the root and its live child rows
+ are purged while the child's history survives, with nothing reported.
+
+ Skipped where nothing is versioned at all. The version cascade is then
+ inert for every root, including this package's own, and refusing would
+ only take those down with it.
+ """
+ try:
+ from sqlalchemy_continuum import version_class, versioning_manager
+ from sqlalchemy_continuum.exc import ClassNotVersioned
+ except ImportError: # pragma: no cover - versioning not installed
+ return
+ if not getattr(versioning_manager, "version_class_map", None):
+ return
+ try:
+ version_class(policy.model)
+ except ClassNotVersioned:
+ raise RuntimeError(
+ f"Purge root {policy.model.__name__} has no version class, so the "
+ "version cascade returns before reaching a declared target; such "
+ "a policy declares none"
+ ) from None
+
+
+def _validate_version_target(
+ policy: PurgeEntityPolicy, dependency: DependencyPolicy, metadata: sa.MetaData
+) -> None:
+ """Reject a version target whose column does not hold the root's id.
+
+ Version cleanup compares the declared column with the root's id directly,
+ so the column has to be root-relative. Shadow tables are generated and may
+ not exist when a policy is validated, so the check reads their live
+ counterpart -- ``<table>_version`` describes ``<table>``, the naming the
+ shadow declarations already rely on -- and accepts the column when it is
+ the root's own primary key or a foreign key into the root.
+
+ A column naming an intermediate owner, a stage id under a workflow root
+ say, matches history belonging to a different root and leaves this one's
+ behind.
+ """
+ root_table: sa.Table = sa.inspect(policy.model).local_table
+ column: str = cast(str, dependency.version_column)
+ shadow_name: str = dependency.key.related_table
+ suffix: str = "_version"
+ live_name: str = (
+ shadow_name[: -len(suffix)] if shadow_name.endswith(suffix) else shadow_name
Review Comment:
When the declared target has no `_version` suffix, `live_name` falls back to the target itself, so a `VERSION_OWNED` dependency naming a live table (say `notes` with `version_column="project_id"`) clears root-relative validation if that column is a foreign key to the root. Version cleanup then resolves the live table and runs `DELETE FROM notes WHERE project_id = ...`, removing rows a `PRESERVE` declaration meant to keep while the real shadow history is untouched. Should a target that is not a `*_version` shadow be rejected here?
##########
superset/commands/deletion_retention/purge_policy.py:
##########
@@ -860,16 +931,443 @@ def declare(
cleanup_permission=cleanup_dataset_permission,
),
}
- return validate_unique_root_policies(registry.values())
+ return tuple(registry.values())
+
+
+def _policy_label(policy: Any) -> str:
+ """A log-safe name for a payload whose shape is not yet established."""
+ model: Any = getattr(policy, "model", None)
+ return str(getattr(model, "__name__", type(model).__name__))
+
+
+def _host_policy_payload(provided: Any) -> tuple[Any, ...]:
+ """Materialize the provider's payload, or reject its shape.
+
+ Any iterable is accepted -- a list, a tuple, ``dict.values()``, a
+ generator -- since the documented contract is a sequence of policies, not
+ one particular container. Strings and bytes are excluded because they
+ iterate into characters, which would read as a sequence of bad members
+ rather than the wrong type. Materialized once: a generator cannot be
+ walked twice.
+
+ An exception raised *while reading* a lazy payload is left to propagate.
+ Walking a generator runs host code just as calling the provider does, so
+ the two are the same kind of failure and earn the same retry; only the
+ shape of a payload that arrived intact is settled for good.
+ """
+ if isinstance(provided, (str, bytes)) or not isinstance(provided, Iterable):
+ logger.error(
+ "purge_policy: %s returned %s; expected an iterable of PurgeEntityPolicy",
+ HOST_POLICIES_CONFIG_KEY,
+ type(provided).__name__,
+ )
+ return ()
+ return tuple(provided)
-@lru_cache(maxsize=None)
-def _validated_purge_policy(model: type[Any]) -> PurgeEntityPolicy:
- """Validate and return one root policy without blocking unrelated roots."""
+def _unqualified_table_conflict(policy: PurgeEntityPolicy) -> str | None:
+ """Return the first table a bare-name lookup would resolve wrongly.
+
+ Discovery records table identities by bare name while a ``MetaData`` keys
+ them by schema, so a declared ``child`` resolves to the default-schema
+ table even where the policy meant ``host.child`` -- and the cleanup would
+ delete rows belonging to unrelated roots. Carrying schema-qualified
+ identities through discovery and execution is the better fix; until then
+ the ambiguous shape is refused rather than silently mis-resolved.
+ """
+ root_table: sa.Table = sa.inspect(policy.model).local_table
+ metadata: sa.MetaData = root_table.metadata
+ keys_by_name: dict[str, list[str]] = {}
+ for table in metadata.tables.values():
+ keys_by_name.setdefault(table.name, []).append(table.key)
+ declared: set[str] = {root_table.name} | {
+ dependency.key.related_table
+ for dependency in policy.dependencies
+ # Version shadows are resolved by bare name as well, in
+ # _entity_version_targets.
+ if dependency.classification in _EXECUTABLE_CLASSIFICATIONS
+ or dependency.classification is DependencyClassification.VERSION_OWNED
+ }
+ for name in sorted(declared):
+ keys: list[str] = keys_by_name.get(name, [])
+ if len(keys) != 1 or keys[0] != name:
+ return name
+ return None
+
+
+def _admitted_host_policy(
+ candidate: Any,
+ builtin_roots: frozenset[type[Any]],
+ builtin_tables: frozenset[str],
+ builtin_types: frozenset[str],
+) -> PurgeEntityPolicy | None:
+ """Return *candidate* if it is a usable host policy, else ``None``.
+
+ Every check runs behind one handler rather than guarding each attribute:
+ the payload is host-supplied, so reading ``model`` or hashing it may
+ itself raise -- and an exception escaping here would break
+ ``purge_policy_registry`` for the built-in roots too, which is the
+ opposite of what this boundary is for.
+ """
try:
- policy: PurgeEntityPolicy = purge_policy_registry()[model]
- except KeyError as ex:
- raise ValueError(f"Unsupported purge model: {model.__name__}") from ex
+ if not isinstance(candidate, PurgeEntityPolicy):
+ logger.error(
+ "purge_policy: %s returned a %s; expected PurgeEntityPolicy",
+ HOST_POLICIES_CONFIG_KEY,
+ type(candidate).__name__,
+ )
+ return None
+ if not isinstance(candidate.model, type):
+ # Everything downstream treats the root as a mapped class:
+ # sa.inspect, the registry index, the duplicate count.
+ logger.error(
+ "purge_policy: host policy root %s is not a class",
+ _policy_label(candidate),
+ )
+ return None
+ if candidate.model in builtin_roots:
+ logger.error(
+ "purge_policy: host policy for built-in root %s ignored",
+ candidate.model.__name__,
+ )
+ return None
+ root_table: sa.Table = sa.inspect(candidate.model).local_table
+ if root_table.key in builtin_tables:
+ # A different class mapped onto a built-in table -- a subclass of
+ # Slice, say -- is not the built-in root by identity, but purging
+ # it runs host cleanup against this package's own rows. It also
+ # joins the soft-delete registry through the mixin, so the
+ # scheduled task would scan it.
+ logger.error(
+ "purge_policy: host policy root %s is mapped onto built-in "
+ "table %r; ignored",
+ candidate.model.__name__,
+ root_table.key,
+ )
+ return None
+ if not isinstance(candidate.entity_type, str) or not candidate.entity_type:
+ # Read as a string by the cascade, the audit record and every log
+ # line that names the root.
+ logger.error(
+ "purge_policy: host policy for %s declares entity_type %r",
+ candidate.model.__name__,
+ candidate.entity_type,
+ )
+ return None
+ if candidate.entity_type in builtin_types:
+ # Core branches on entity_type -- the dataset impact path, the tag
+ # object type, the dataset permission name -- so a host reusing
+ # one of its names would have that logic pointed at its own rows.
+ logger.error(
+ "purge_policy: host policy for %s claims the reserved entity type %r",
+ candidate.model.__name__,
+ candidate.entity_type,
+ )
+ return None
+ # Validated first, so a table the metadata simply does not contain is
+ # reported as such rather than as an ambiguous name.
+ unusable: set[str] = {
+ dependency.classification.value
+ for dependency in candidate.dependencies
+ if dependency.classification
+ in {
+ DependencyClassification.BLOCK,
+ DependencyClassification.LISTENER_EFFECT,
+ }
+ }
+ if unusable:
+ # Two different reasons, both about identities this package owns.
+ # A stock listener action decides what to delete from a built-in
+ # entity type, so declared by a host it is accepted and then never
+ # runs. A blocker *would* be applied -- validate_deletion_allowed
+ # is generic over the declarations -- but its reason code lands in
+ # the purge audit record, whose vocabulary is a closed set pinned
+ # by test. A host refuses a purge from its own validator instead,
+ # where the code it records is plainly its own.
+ raise RuntimeError(
+ f"{', '.join(sorted(unusable))} dependencies are not available "
+ "to a host root"
+ )
+ _validated_policy(candidate)
+ conflict: str | None = _unqualified_table_conflict(candidate)
+ if conflict is not None:
+ raise RuntimeError(
+ f"table {conflict!r} cannot be identified unambiguously by "
+ "name; schema-qualified roots are not supported"
+ )
+ except Exception as ex: # pylint: disable=broad-except
+ # Validated at admission rather than when a row is purged: a
+ # declaration the cleanup cannot execute would otherwise fail once per
+ # eligible row, counted as a cascade failure, and stay invisible until
+ # something aged past the window.
+ logger.error(
+ "purge_policy: host policy for %s rejected: %s",
+ _policy_label(candidate),
+ ex,
+ )
+ return None
+ return candidate
+
+
+def _host_purge_policies(
+ provider: Callable[[], Any] | None,
+) -> tuple[PurgeEntityPolicy, ...] | None:
+ """Return the purge policies a host installed for its own roots.
+
+ A host distribution can carry ``SoftDeleteMixin`` entities this package
+ cannot import. The retention task discovers those entities through the
+ mixin registry, so without a policy they reach the cascade as an
+ unsupported model. The host therefore declares their purge behavior and
+ installs it under ``PURGE_POLICIES_FUNC``.
+
+ Host boundary: everything about a host policy is settled here, where it is
+ admitted -- an unavailable provider, a payload that is not an iterable of
+ policies, a root that is not a class, a collision with a root or entity
+ type declared in this package, two declarations for one root, and a
+ declaration the shared cleanup could not execute. Each is logged and
+ dropped, so that root is reported as unsupported instead of failing row by
+ row in a scheduled run. A broken host declaration must not take the purge
+ down with it, and must never redefine how a chart, dashboard or dataset is
+ purged.
+
+ Returns ``None`` for the failures that may be transient -- the provider
+ could not be called, or a lazy payload raised while being read. The caller
+ then publishes the built-in roots with a short deadline, so the next
+ resolution after it tries again rather than treating one bad moment as the
+ answer for the life of the process. A payload that arrived intact but is
+ the wrong shape is a host bug that will not fix itself, so it is published
+ as an empty result instead.
+ """
+ if provider is None:
+ return ()
+ if not callable(provider):
+ # Static misconfiguration, not an outage: reported once rather than
+ # retried with a traceback for the life of the process.
+ logger.error(
+ "purge_policy: %s is %s, not a callable returning policies",
+ HOST_POLICIES_CONFIG_KEY,
+ type(provider).__name__,
+ )
+ return ()
+ try:
+ # Both the call and reading what it returns run host code, so a lazy
+ # payload that raises mid-walk is the same kind of failure as a
+ # provider that raises outright, and earns the same retry.
+ candidates: tuple[Any, ...] = _host_policy_payload(provider())
+ except Exception: # pylint: disable=broad-except
+ logger.exception(
+ "purge_policy: %s is unavailable; keeping built-in roots only",
+ HOST_POLICIES_CONFIG_KEY,
+ )
+ return None
+
+ builtin_policies: tuple[PurgeEntityPolicy, ...] = _builtin_purge_policies()
+ builtin_roots: frozenset[type[Any]] = frozenset(
+ policy.model for policy in builtin_policies
+ )
+ builtin_tables: frozenset[str] = frozenset(
+ sa.inspect(policy.model).local_table.key for policy in builtin_policies
+ )
+ builtin_types: frozenset[str] = frozenset(
+ policy.entity_type for policy in builtin_policies
+ )
+ admitted: list[PurgeEntityPolicy] = [
+ policy
+ for candidate in candidates
+ if (
+ policy := _admitted_host_policy(
+ candidate, builtin_roots, builtin_tables, builtin_types
+ )
+ )
+ is not None
+ ]
+
+ # Counted after admission, where every root is known to be a class and so
+ # hashable. Which of two declarations for one root is authoritative is
+ # undecidable, and the loser would still delete rows, so both are dropped:
+ # the root is reported as unsupported, which is the recoverable outcome.
+ # It also keeps the duplicate away from validate_unique_root_policies,
+ # whose ValueError would abort the whole scheduled run.
+ counts: dict[type[Any], int] = {}
+ for policy in admitted:
+ counts[policy.model] = counts.get(policy.model, 0) + 1
+ for model, count in counts.items():
+ if count > 1:
+ logger.error(
+ "purge_policy: %s declares %d policies for %s; ignoring all of them",
+ HOST_POLICIES_CONFIG_KEY,
+ count,
+ model.__name__,
+ )
+ return tuple(policy for policy in admitted if counts[policy.model] == 1)
+
+
+#: Sentinel distinguishing "no index resolved yet" from an index resolved for
+#: no provider, which is the ordinary case.
+_UNRESOLVED: Any = object()
+
+
+@dataclass
+class _ResolvedRegistry:
+ """The index built for one provider, with the roots validated so far.
+
+ Matched to its provider by *identity* rather than keyed in an
+ ``lru_cache``: a cache key would have to be hashed, and a host callable
+ implemented as a mutable dataclass instance is unhashable. That raises
+ before the host boundary can isolate the host's mistake, which would
+ stop the built-in roots purging too.
+ """
+
+ provider: Any = _UNRESOLVED
+ registry: Mapping[type[Any], PurgeEntityPolicy] = field(
+ default_factory=lambda: MappingProxyType({})
+ )
+ validated: set[type[Any]] = field(default_factory=set)
+ #: Set only on a snapshot published because the provider could not be
+ #: called. Until this deadline the snapshot answers reads; after it, the
+ #: provider is tried again.
+ retry_after: float | None = None
+
+
+#: Key the resolved index lives under on the Flask app. Per app rather than
+#: per process: two apps in one process have different configs, and a single
+#: global snapshot would thrash between them -- re-resolving, and re-invoking
+#: the provider, on every read.
+_REGISTRY_EXTENSION_KEY: str = "deletion_retention_purge_registry"
+
+#: How long a snapshot published after a provider failure answers reads
+#: before the provider is tried again. Bounds both extremes: a failure is
+#: neither remembered for the life of the process nor retried on every read --
+#: at roughly three reads per purged entity, retrying per read meant tens of
+#: thousands of provider calls and tracebacks in a single pass.
+_PROVIDER_RETRY_SECONDS: float = 60.0
+
+#: Per-thread marker for "this thread is already resolving". A host provider
+#: may build its policy from a built-in one -- ``replace(get_purge_policy(
+#: Slice), model=HostRoot, ...)`` is the obvious way to borrow the stock
+#: callbacks -- which re-enters resolution while the first call is still in
+#: the provider. The nested call is handed the state already published
+#: instead, which always carries the built-in roots.
+_RESOLVING: threading.local = threading.local()
+
+
+def _resolved_registry() -> _ResolvedRegistry:
+ """Return the state resolved for the installed provider, building it once.
+
+ Readers take a snapshot and a rebuild publishes a new one, so a caller
+ always sees an index and its validation record together. Publishing is a
+ single rebind, which is what makes it safe to do this without a lock: two
+ concurrent first uses may each invoke the provider and each publish, but
+ the two snapshots are equivalent and neither is ever seen half-built.
+ Paying for a duplicate provider call is the deliberate trade -- a lock
+ here deadlocks the moment a provider resolves a built-in policy of its
+ own, and a hung purge is worse than a repeated call.
+ """
+ if not has_app_context():
+ return _builtin_registry()
+ provider: Callable[[], Any] | None = current_app.config.get(
+ HOST_POLICIES_CONFIG_KEY
+ )
+ published: _ResolvedRegistry | None = current_app.extensions.get(
+ _REGISTRY_EXTENSION_KEY
+ )
+ if published is None:
+ published = _builtin_registry()
+ current_app.extensions[_REGISTRY_EXTENSION_KEY] = published
+ if published.provider is provider and (
+ published.retry_after is None or time.monotonic() < published.retry_after
+ ):
+ return published
+ if getattr(_RESOLVING, "active", False):
+ # A nested resolution: the provider is building its policy from a
+ # built-in one, so hand it the roots this package declares rather than
+ # re-entering a resolution that has not finished.
+ return _builtin_registry()
+ # Not nested: the branch above returns when the marker is already set,
+ # so reaching here means this thread was not resolving.
+ _RESOLVING.active = True
+ try:
+ host_policies: tuple[PurgeEntityPolicy, ...] | None = _host_purge_policies(
+ provider
+ )
+ finally:
+ _RESOLVING.active = False
+ if host_policies is None:
+ current: _ResolvedRegistry | None = current_app.extensions.get(
+ _REGISTRY_EXTENSION_KEY
+ )
+ if (
+ current is not None
+ and current.provider is provider
+ and current.retry_after is None
Review Comment:
This branch keeps a good snapshot when a slower caller fails after a faster one published, but no test pins that ordering: the existing concurrent test has both calls succeed, and the sequential retry tests never reach it. If the guard were removed, the late failure would overwrite the host index with built-ins only and host roots would report as unsupported for the retry window. Could a registry unit test use events to publish a successful resolution before a second call's provider raises, then assert both that caller and a later lookup still see the host policy without another provider call?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(retention): let a host install purge policies for its own soft-delete roots [superset]
Posted by "alexandrusoare (via GitHub)" <gi...@apache.org>.
alexandrusoare commented on code in PR #44892:
URL: https://github.com/apache/superset/pull/44892#discussion_r4183363293
##########
superset/commands/deletion_retention/purge_policy.py:
##########
@@ -860,7 +867,96 @@ def declare(
cleanup_permission=cleanup_dataset_permission,
),
}
- return validate_unique_root_policies(registry.values())
+ return tuple(registry.values())
+
+
+def _host_purge_policies() -> tuple[PurgeEntityPolicy, ...]:
+ """Return the purge policies a host installed for its own roots.
+
+ A host distribution can carry ``SoftDeleteMixin`` entities this package
+ cannot import. The retention task discovers those entities through the
+ mixin registry, so without a policy they reach the cascade as an
+ unsupported model. The host therefore declares their purge behavior and
+ installs it under ``PURGE_POLICIES_FUNC``.
+
+ Host boundary: an unavailable provider, a malformed payload, a policy that
+ collides with a root declared here, and two policies for one host root are
+ each logged and dropped. A broken host declaration must not take the
+ scheduled purge down with it, and must never redefine how a chart,
+ dashboard or dataset is purged.
+ """
+ if not has_app_context():
+ return ()
+ provider: Callable[[], Any] | None = current_app.config.get(
+ HOST_POLICIES_CONFIG_KEY
+ )
+ if provider is None:
+ return ()
+ try:
+ provided: Any = provider()
+ except Exception: # pylint: disable=broad-except
+ logger.exception(
+ "purge_policy: %s is unavailable; keeping built-in roots only",
+ HOST_POLICIES_CONFIG_KEY,
+ )
+ return ()
+ if not isinstance(provided, (list, tuple)) or not all(
+ isinstance(policy, PurgeEntityPolicy) for policy in provided
+ ):
+ logger.error(
+ "purge_policy: %s returned %s; expected a sequence of PurgeEntityPolicy",
+ HOST_POLICIES_CONFIG_KEY,
+ type(provided).__name__,
+ )
+ return ()
+ builtin_roots: frozenset[type[Any]] = frozenset(
+ policy.model for policy in _builtin_purge_policies()
+ )
+ declared: dict[type[Any], int] = {}
+ for policy in provided:
+ declared[policy.model] = declared.get(policy.model, 0) + 1
+ # Which of two declarations for one root is authoritative is undecidable,
+ # and the loser would still delete rows. Dropping both leaves the model
+ # reported as unsupported, which is the recoverable outcome. Resolving it
+ # here also keeps the duplicate away from validate_unique_root_policies,
+ # whose ValueError would abort the whole scheduled run.
+ duplicated: list[type[Any]] = [
+ model for model, count in declared.items() if count > 1
+ ]
+ for model in duplicated:
+ logger.error(
+ "purge_policy: %s declares %d policies for %s; ignoring all of them",
+ HOST_POLICIES_CONFIG_KEY,
+ declared[model],
+ model.__name__,
+ )
+ accepted: list[PurgeEntityPolicy] = []
+ for policy in provided:
+ if policy.model in builtin_roots:
+ logger.error(
+ "purge_policy: host policy for built-in root %s ignored",
+ policy.model.__name__,
+ )
+ continue
+ if policy.model in duplicated:
+ continue
+ accepted.append(policy)
+ return tuple(accepted)
+
+
+def purge_policy_registry() -> Mapping[type[Any], PurgeEntityPolicy]:
+ """Index the built-in purge roots plus any the host installed.
+
+ Uncached on purpose, unlike its two inputs: the built-in declarations are
+ built once per process, while a host policy is resolved per call, so a
+ provider installed after the first purge is honored for any root not yet
+ resolved. ``get_purge_policy`` memoizes per model, so replacing the policy
+ of a root it has already resolved needs a restart -- and that memoization
+ is why rebuilding this small index is off the hot path.
+ """
+ return validate_unique_root_policies(
Review Comment:
Good catch — visited restarted per branch, so every sibling re-walked all the others. Now tracked across the whole traversal, with a test pinning the count.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(retention): let a host install purge policies for its own soft-delete roots [superset]
Posted by "alexandrusoare (via GitHub)" <gi...@apache.org>.
alexandrusoare commented on code in PR #44892:
URL: https://github.com/apache/superset/pull/44892#discussion_r4183357165
##########
superset/commands/deletion_retention/purge_policy.py:
##########
@@ -860,7 +867,96 @@ def declare(
cleanup_permission=cleanup_dataset_permission,
),
}
- return validate_unique_root_policies(registry.values())
+ return tuple(registry.values())
+
+
+def _host_purge_policies() -> tuple[PurgeEntityPolicy, ...]:
+ """Return the purge policies a host installed for its own roots.
+
+ A host distribution can carry ``SoftDeleteMixin`` entities this package
+ cannot import. The retention task discovers those entities through the
+ mixin registry, so without a policy they reach the cascade as an
+ unsupported model. The host therefore declares their purge behavior and
+ installs it under ``PURGE_POLICIES_FUNC``.
+
+ Host boundary: an unavailable provider, a malformed payload, a policy that
+ collides with a root declared here, and two policies for one host root are
+ each logged and dropped. A broken host declaration must not take the
+ scheduled purge down with it, and must never redefine how a chart,
+ dashboard or dataset is purged.
+ """
+ if not has_app_context():
+ return ()
+ provider: Callable[[], Any] | None = current_app.config.get(
+ HOST_POLICIES_CONFIG_KEY
+ )
+ if provider is None:
+ return ()
+ try:
+ provided: Any = provider()
+ except Exception: # pylint: disable=broad-except
+ logger.exception(
+ "purge_policy: %s is unavailable; keeping built-in roots only",
+ HOST_POLICIES_CONFIG_KEY,
+ )
+ return ()
+ if not isinstance(provided, (list, tuple)) or not all(
+ isinstance(policy, PurgeEntityPolicy) for policy in provided
+ ):
+ logger.error(
+ "purge_policy: %s returned %s; expected a sequence of PurgeEntityPolicy",
+ HOST_POLICIES_CONFIG_KEY,
+ type(provided).__name__,
+ )
+ return ()
+ builtin_roots: frozenset[type[Any]] = frozenset(
+ policy.model for policy in _builtin_purge_policies()
+ )
+ declared: dict[type[Any], int] = {}
+ for policy in provided:
+ declared[policy.model] = declared.get(policy.model, 0) + 1
+ # Which of two declarations for one root is authoritative is undecidable,
+ # and the loser would still delete rows. Dropping both leaves the model
+ # reported as unsupported, which is the recoverable outcome. Resolving it
+ # here also keeps the duplicate away from validate_unique_root_policies,
+ # whose ValueError would abort the whole scheduled run.
+ duplicated: list[type[Any]] = [
+ model for model, count in declared.items() if count > 1
+ ]
+ for model in duplicated:
+ logger.error(
+ "purge_policy: %s declares %d policies for %s; ignoring all of them",
+ HOST_POLICIES_CONFIG_KEY,
+ declared[model],
+ model.__name__,
+ )
+ accepted: list[PurgeEntityPolicy] = []
+ for policy in provided:
+ if policy.model in builtin_roots:
+ logger.error(
+ "purge_policy: host policy for built-in root %s ignored",
+ policy.model.__name__,
+ )
+ continue
+ if policy.model in duplicated:
+ continue
+ accepted.append(policy)
Review Comment:
Confirmed, and it took the whole run down with it. Took both your options: registration now requires id and deleted_at on the root, and the task handles each root separately — a scan that raises is counted in scan_failures and the rest still purge.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(retention): let a host install purge policies for its own soft-delete roots [superset]
Posted by "alexandrusoare (via GitHub)" <gi...@apache.org>.
alexandrusoare commented on code in PR #44892:
URL: https://github.com/apache/superset/pull/44892#discussion_r4183354563
##########
superset/commands/deletion_retention/purge_policy.py:
##########
@@ -909,6 +1005,58 @@ def _validated_purge_policy(model: type[Any]) -> PurgeEntityPolicy:
return policy
+def _ownership_dependency(
+ policy: PurgeEntityPolicy, related_table: str
+) -> DependencyPolicy | None:
+ """The single owned/association edge attaching *related_table*, if clear."""
+ candidates: tuple[DependencyPolicy, ...] = tuple(
+ dependency
+ for dependency in policy.dependencies
+ if dependency.classification
+ in {DependencyClassification.OWNED, DependencyClassification.ASSOCIATION}
+ and dependency.key.related_table == related_table
+ and dependency.key.direction == "inbound"
+ )
+ return candidates[0] if len(candidates) == 1 else None
+
+
+def _validate_owned_traversal(policy: PurgeEntityPolicy) -> None:
+ """Reject an owned table reachable only through an association.
+
+ ``cascade_hard_delete`` empties associations before owned children, while
+ an owned table's predicate selects its rows *through* its ownership path
+ (see ``_owner_value_select``). If a hop on that path is an association,
+ its rows are already gone when the owned delete runs: the statement
+ matches nothing, leaving the descendants orphaned where foreign keys are
+ unenforced and blocking the root's delete where they are not.
+
+ Refused at declaration time rather than executed. The shape has a
+ remedy -- classify the intermediate table as owned, which places it in
+ the same phase as what it leads to.
+ """
+ root_table: str = sa.inspect(policy.model).local_table.name
+ for dependency in policy.dependencies:
+ if dependency.classification is not DependencyClassification.OWNED:
+ continue
+ table_name: str = dependency.key.owner_table
+ visited: set[str] = set()
+ while table_name != root_table and table_name not in visited:
Review Comment:
Right — the stock cleanup deletes one level, so a deeper tree gets left behind. Validation now refuses a self-referencing owned table unless the policy brings its own delete_owned_children.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(retention): let a host install purge policies for its own soft-delete roots [superset]
Posted by "alexandrusoare (via GitHub)" <gi...@apache.org>.
alexandrusoare commented on code in PR #44892:
URL: https://github.com/apache/superset/pull/44892#discussion_r4183363293
##########
superset/commands/deletion_retention/purge_policy.py:
##########
@@ -860,7 +867,96 @@ def declare(
cleanup_permission=cleanup_dataset_permission,
),
}
- return validate_unique_root_policies(registry.values())
+ return tuple(registry.values())
+
+
+def _host_purge_policies() -> tuple[PurgeEntityPolicy, ...]:
+ """Return the purge policies a host installed for its own roots.
+
+ A host distribution can carry ``SoftDeleteMixin`` entities this package
+ cannot import. The retention task discovers those entities through the
+ mixin registry, so without a policy they reach the cascade as an
+ unsupported model. The host therefore declares their purge behavior and
+ installs it under ``PURGE_POLICIES_FUNC``.
+
+ Host boundary: an unavailable provider, a malformed payload, a policy that
+ collides with a root declared here, and two policies for one host root are
+ each logged and dropped. A broken host declaration must not take the
+ scheduled purge down with it, and must never redefine how a chart,
+ dashboard or dataset is purged.
+ """
+ if not has_app_context():
+ return ()
+ provider: Callable[[], Any] | None = current_app.config.get(
+ HOST_POLICIES_CONFIG_KEY
+ )
+ if provider is None:
+ return ()
+ try:
+ provided: Any = provider()
+ except Exception: # pylint: disable=broad-except
+ logger.exception(
+ "purge_policy: %s is unavailable; keeping built-in roots only",
+ HOST_POLICIES_CONFIG_KEY,
+ )
+ return ()
+ if not isinstance(provided, (list, tuple)) or not all(
+ isinstance(policy, PurgeEntityPolicy) for policy in provided
+ ):
+ logger.error(
+ "purge_policy: %s returned %s; expected a sequence of PurgeEntityPolicy",
+ HOST_POLICIES_CONFIG_KEY,
+ type(provided).__name__,
+ )
+ return ()
+ builtin_roots: frozenset[type[Any]] = frozenset(
+ policy.model for policy in _builtin_purge_policies()
+ )
+ declared: dict[type[Any], int] = {}
+ for policy in provided:
+ declared[policy.model] = declared.get(policy.model, 0) + 1
+ # Which of two declarations for one root is authoritative is undecidable,
+ # and the loser would still delete rows. Dropping both leaves the model
+ # reported as unsupported, which is the recoverable outcome. Resolving it
+ # here also keeps the duplicate away from validate_unique_root_policies,
+ # whose ValueError would abort the whole scheduled run.
+ duplicated: list[type[Any]] = [
+ model for model, count in declared.items() if count > 1
+ ]
+ for model in duplicated:
+ logger.error(
+ "purge_policy: %s declares %d policies for %s; ignoring all of them",
+ HOST_POLICIES_CONFIG_KEY,
+ declared[model],
+ model.__name__,
+ )
+ accepted: list[PurgeEntityPolicy] = []
+ for policy in provided:
+ if policy.model in builtin_roots:
+ logger.error(
+ "purge_policy: host policy for built-in root %s ignored",
+ policy.model.__name__,
+ )
+ continue
+ if policy.model in duplicated:
+ continue
+ accepted.append(policy)
+ return tuple(accepted)
+
+
+def purge_policy_registry() -> Mapping[type[Any], PurgeEntityPolicy]:
+ """Index the built-in purge roots plus any the host installed.
+
+ Uncached on purpose, unlike its two inputs: the built-in declarations are
+ built once per process, while a host policy is resolved per call, so a
+ provider installed after the first purge is honored for any root not yet
+ resolved. ``get_purge_policy`` memoizes per model, so replacing the policy
+ of a root it has already resolved needs a restart -- and that memoization
+ is why rebuilding this small index is off the hot path.
+ """
+ return validate_unique_root_policies(
Review Comment:
Good catch — visited restarted per branch, so every sibling re-walked all the others. Now tracked across the whole traversal, with a test pinning the count. One subtlety: a mapper walk sees relationship edges a plain table walk can't, so the two are tracked separately — merging them dropped sql_metrics → sql_metrics_version from the dataset policy, which the existing tests caught.
##########
superset/commands/deletion_retention/purge_policy.py:
##########
@@ -926,6 +1074,7 @@ def _validate_executable_declarations(policy: PurgeEntityPolicy) -> None:
raise RuntimeError(
f"Missing version target column for {dependency.key.describe()}"
)
+ _validate_owned_traversal(policy)
Review Comment:
Agreed. Went with your first option: two same-depth owned tables that reference each other are now rejected.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(retention): let a host install purge policies for its own soft-delete roots [superset]
Posted by "alexandrusoare (via GitHub)" <gi...@apache.org>.
alexandrusoare commented on code in PR #44892:
URL: https://github.com/apache/superset/pull/44892#discussion_r4183363293
##########
superset/commands/deletion_retention/purge_policy.py:
##########
@@ -860,7 +867,96 @@ def declare(
cleanup_permission=cleanup_dataset_permission,
),
}
- return validate_unique_root_policies(registry.values())
+ return tuple(registry.values())
+
+
+def _host_purge_policies() -> tuple[PurgeEntityPolicy, ...]:
+ """Return the purge policies a host installed for its own roots.
+
+ A host distribution can carry ``SoftDeleteMixin`` entities this package
+ cannot import. The retention task discovers those entities through the
+ mixin registry, so without a policy they reach the cascade as an
+ unsupported model. The host therefore declares their purge behavior and
+ installs it under ``PURGE_POLICIES_FUNC``.
+
+ Host boundary: an unavailable provider, a malformed payload, a policy that
+ collides with a root declared here, and two policies for one host root are
+ each logged and dropped. A broken host declaration must not take the
+ scheduled purge down with it, and must never redefine how a chart,
+ dashboard or dataset is purged.
+ """
+ if not has_app_context():
+ return ()
+ provider: Callable[[], Any] | None = current_app.config.get(
+ HOST_POLICIES_CONFIG_KEY
+ )
+ if provider is None:
+ return ()
+ try:
+ provided: Any = provider()
+ except Exception: # pylint: disable=broad-except
+ logger.exception(
+ "purge_policy: %s is unavailable; keeping built-in roots only",
+ HOST_POLICIES_CONFIG_KEY,
+ )
+ return ()
+ if not isinstance(provided, (list, tuple)) or not all(
+ isinstance(policy, PurgeEntityPolicy) for policy in provided
+ ):
+ logger.error(
+ "purge_policy: %s returned %s; expected a sequence of PurgeEntityPolicy",
+ HOST_POLICIES_CONFIG_KEY,
+ type(provided).__name__,
+ )
+ return ()
+ builtin_roots: frozenset[type[Any]] = frozenset(
+ policy.model for policy in _builtin_purge_policies()
+ )
+ declared: dict[type[Any], int] = {}
+ for policy in provided:
+ declared[policy.model] = declared.get(policy.model, 0) + 1
+ # Which of two declarations for one root is authoritative is undecidable,
+ # and the loser would still delete rows. Dropping both leaves the model
+ # reported as unsupported, which is the recoverable outcome. Resolving it
+ # here also keeps the duplicate away from validate_unique_root_policies,
+ # whose ValueError would abort the whole scheduled run.
+ duplicated: list[type[Any]] = [
+ model for model, count in declared.items() if count > 1
+ ]
+ for model in duplicated:
+ logger.error(
+ "purge_policy: %s declares %d policies for %s; ignoring all of them",
+ HOST_POLICIES_CONFIG_KEY,
+ declared[model],
+ model.__name__,
+ )
+ accepted: list[PurgeEntityPolicy] = []
+ for policy in provided:
+ if policy.model in builtin_roots:
+ logger.error(
+ "purge_policy: host policy for built-in root %s ignored",
+ policy.model.__name__,
+ )
+ continue
+ if policy.model in duplicated:
+ continue
+ accepted.append(policy)
+ return tuple(accepted)
+
+
+def purge_policy_registry() -> Mapping[type[Any], PurgeEntityPolicy]:
+ """Index the built-in purge roots plus any the host installed.
+
+ Uncached on purpose, unlike its two inputs: the built-in declarations are
+ built once per process, while a host policy is resolved per call, so a
+ provider installed after the first purge is honored for any root not yet
+ resolved. ``get_purge_policy`` memoizes per model, so replacing the policy
+ of a root it has already resolved needs a restart -- and that memoization
+ is why rebuilding this small index is off the hot path.
+ """
+ return validate_unique_root_policies(
Review Comment:
Good catch — visited restarted per branch, so every sibling re-walked all the others. Now tracked across the whole traversal, with a test pinning the count. One subtlety: a mapper walk sees relationship edges a plain table walk can't, so the two are tracked separately
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(retention): let a host install purge policies for its own soft-delete roots [superset]
Posted by "msyavuz (via GitHub)" <gi...@apache.org>.
msyavuz commented on code in PR #44892:
URL: https://github.com/apache/superset/pull/44892#discussion_r4195697722
##########
superset/commands/deletion_retention/purge_policy.py:
##########
@@ -860,16 +914,396 @@ def declare(
cleanup_permission=cleanup_dataset_permission,
),
}
- return validate_unique_root_policies(registry.values())
+ return tuple(registry.values())
+
+
+def _policy_label(policy: Any) -> str:
+ """A log-safe name for a payload whose shape is not yet established."""
+ model: Any = getattr(policy, "model", None)
+ return str(getattr(model, "__name__", type(model).__name__))
+
+
+def _host_policy_payload(provided: Any) -> tuple[Any, ...]:
+ """Materialize the provider's payload, or reject its shape.
+
+ Any iterable is accepted -- a list, a tuple, ``dict.values()``, a
+ generator -- since the documented contract is a sequence of policies, not
+ one particular container. Strings and bytes are excluded because they
+ iterate into characters, which would read as a sequence of bad members
+ rather than the wrong type. Materialized once: a generator cannot be
+ walked twice.
+
+ An exception raised *while reading* a lazy payload is left to propagate.
+ Walking a generator runs host code just as calling the provider does, so
+ the two are the same kind of failure and earn the same retry; only the
+ shape of a payload that arrived intact is settled for good.
+ """
+ if isinstance(provided, (str, bytes)) or not isinstance(provided, Iterable):
+ logger.error(
+ "purge_policy: %s returned %s; expected an iterable of PurgeEntityPolicy",
+ HOST_POLICIES_CONFIG_KEY,
+ type(provided).__name__,
+ )
+ return ()
+ return tuple(provided)
+
+def _unqualified_table_conflict(policy: PurgeEntityPolicy) -> str | None:
+ """Return the first table a bare-name lookup would resolve wrongly.
-@lru_cache(maxsize=None)
-def _validated_purge_policy(model: type[Any]) -> PurgeEntityPolicy:
- """Validate and return one root policy without blocking unrelated roots."""
+ Discovery records table identities by bare name while a ``MetaData`` keys
+ them by schema, so a declared ``child`` resolves to the default-schema
+ table even where the policy meant ``host.child`` -- and the cleanup would
+ delete rows belonging to unrelated roots. Carrying schema-qualified
+ identities through discovery and execution is the better fix; until then
+ the ambiguous shape is refused rather than silently mis-resolved.
+ """
+ root_table: sa.Table = sa.inspect(policy.model).local_table
+ metadata: sa.MetaData = root_table.metadata
+ keys_by_name: dict[str, list[str]] = {}
+ for table in metadata.tables.values():
+ keys_by_name.setdefault(table.name, []).append(table.key)
+ declared: set[str] = {root_table.name} | {
+ dependency.key.related_table
+ for dependency in policy.dependencies
+ # Version shadows are resolved by bare name as well, in
+ # _entity_version_targets.
+ if dependency.classification in _EXECUTABLE_CLASSIFICATIONS
+ or dependency.classification is DependencyClassification.VERSION_OWNED
+ }
+ for name in sorted(declared):
+ keys: list[str] = keys_by_name.get(name, [])
+ if len(keys) != 1 or keys[0] != name:
+ return name
+ return None
+
+
+def _admitted_host_policy(
+ candidate: Any, builtin_roots: frozenset[type[Any]], builtin_types: frozenset[str]
+) -> PurgeEntityPolicy | None:
+ """Return *candidate* if it is a usable host policy, else ``None``.
+
+ Every check runs behind one handler rather than guarding each attribute:
+ the payload is host-supplied, so reading ``model`` or hashing it may
+ itself raise -- and an exception escaping here would break
+ ``purge_policy_registry`` for the built-in roots too, which is the
+ opposite of what this boundary is for.
+ """
try:
- policy: PurgeEntityPolicy = purge_policy_registry()[model]
- except KeyError as ex:
- raise ValueError(f"Unsupported purge model: {model.__name__}") from ex
+ if not isinstance(candidate, PurgeEntityPolicy):
+ logger.error(
+ "purge_policy: %s returned a %s; expected PurgeEntityPolicy",
+ HOST_POLICIES_CONFIG_KEY,
+ type(candidate).__name__,
+ )
+ return None
+ if not isinstance(candidate.model, type):
+ # Everything downstream treats the root as a mapped class:
+ # sa.inspect, the registry index, the duplicate count.
+ logger.error(
+ "purge_policy: host policy root %s is not a class",
+ _policy_label(candidate),
+ )
+ return None
+ if candidate.model in builtin_roots:
+ logger.error(
+ "purge_policy: host policy for built-in root %s ignored",
+ candidate.model.__name__,
+ )
+ return None
+ if candidate.entity_type in builtin_types:
+ # Core branches on entity_type -- the dataset impact path, the tag
+ # object type, the dataset permission name -- so a host reusing
+ # one of its names would have that logic pointed at its own rows.
+ logger.error(
+ "purge_policy: host policy for %s claims the reserved entity type %r",
+ candidate.model.__name__,
+ candidate.entity_type,
+ )
+ return None
+ # Validated first, so a table the metadata simply does not contain is
+ # reported as such rather than as an ambiguous name.
+ unusable: set[str] = {
+ dependency.classification.value
+ for dependency in candidate.dependencies
+ if dependency.classification
+ in {
+ DependencyClassification.BLOCK,
+ DependencyClassification.LISTENER_EFFECT,
+ }
+ }
+ if unusable:
+ # Two different reasons, both about identities this package owns.
+ # A stock listener action decides what to delete from a built-in
+ # entity type, so declared by a host it is accepted and then never
+ # runs. A blocker *would* be applied -- validate_deletion_allowed
+ # is generic over the declarations -- but its reason code lands in
+ # the purge audit record, whose vocabulary is a closed set pinned
+ # by test. A host refuses a purge from its own validator instead,
+ # where the code it records is plainly its own.
+ raise RuntimeError(
+ f"{', '.join(sorted(unusable))} dependencies are not available "
+ "to a host root"
+ )
+ _validated_policy(candidate)
+ conflict: str | None = _unqualified_table_conflict(candidate)
+ if conflict is not None:
+ raise RuntimeError(
+ f"table {conflict!r} cannot be identified unambiguously by "
+ "name; schema-qualified roots are not supported"
+ )
+ except Exception as ex: # pylint: disable=broad-except
+ # Validated at admission rather than when a row is purged: a
+ # declaration the cleanup cannot execute would otherwise fail once per
+ # eligible row, counted as a cascade failure, and stay invisible until
+ # something aged past the window.
+ logger.error(
+ "purge_policy: host policy for %s rejected: %s",
+ _policy_label(candidate),
+ ex,
+ )
+ return None
+ return candidate
+
+
+def _host_purge_policies(
+ provider: Callable[[], Any] | None,
+) -> tuple[PurgeEntityPolicy, ...] | None:
+ """Return the purge policies a host installed for its own roots.
+
+ A host distribution can carry ``SoftDeleteMixin`` entities this package
+ cannot import. The retention task discovers those entities through the
+ mixin registry, so without a policy they reach the cascade as an
+ unsupported model. The host therefore declares their purge behavior and
+ installs it under ``PURGE_POLICIES_FUNC``.
+
+ Host boundary: everything about a host policy is settled here, where it is
+ admitted -- an unavailable provider, a payload that is not an iterable of
+ policies, a root that is not a class, a collision with a root or entity
+ type declared in this package, two declarations for one root, and a
+ declaration the shared cleanup could not execute. Each is logged and
+ dropped, so that root is reported as unsupported instead of failing row by
+ row in a scheduled run. A broken host declaration must not take the purge
+ down with it, and must never redefine how a chart, dashboard or dataset is
+ purged.
+
+ Returns ``None`` for the failures that may be transient -- the provider
+ could not be called, or a lazy payload raised while being read. The caller
+ then publishes the built-in roots with a short deadline, so the next
+ resolution after it tries again rather than treating one bad moment as the
+ answer for the life of the process. A payload that arrived intact but is
+ the wrong shape is a host bug that will not fix itself, so it is published
+ as an empty result instead.
+ """
+ if provider is None:
+ return ()
+ try:
+ # Both the call and reading what it returns run host code, so a lazy
+ # payload that raises mid-walk is the same kind of failure as a
+ # provider that raises outright, and earns the same retry.
+ candidates: tuple[Any, ...] = _host_policy_payload(provider())
+ except Exception: # pylint: disable=broad-except
+ logger.exception(
+ "purge_policy: %s is unavailable; keeping built-in roots only",
+ HOST_POLICIES_CONFIG_KEY,
+ )
+ return None
+
+ builtin_policies: tuple[PurgeEntityPolicy, ...] = _builtin_purge_policies()
+ builtin_roots: frozenset[type[Any]] = frozenset(
+ policy.model for policy in builtin_policies
+ )
+ builtin_types: frozenset[str] = frozenset(
+ policy.entity_type for policy in builtin_policies
+ )
+ admitted: list[PurgeEntityPolicy] = [
+ policy
+ for candidate in candidates
+ if (policy := _admitted_host_policy(candidate, builtin_roots, builtin_types))
+ is not None
+ ]
+
+ # Counted after admission, where every root is known to be a class and so
+ # hashable. Which of two declarations for one root is authoritative is
+ # undecidable, and the loser would still delete rows, so both are dropped:
+ # the root is reported as unsupported, which is the recoverable outcome.
+ # It also keeps the duplicate away from validate_unique_root_policies,
+ # whose ValueError would abort the whole scheduled run.
+ counts: dict[type[Any], int] = {}
+ for policy in admitted:
+ counts[policy.model] = counts.get(policy.model, 0) + 1
+ for model, count in counts.items():
+ if count > 1:
+ logger.error(
+ "purge_policy: %s declares %d policies for %s; ignoring all of them",
+ HOST_POLICIES_CONFIG_KEY,
+ count,
+ model.__name__,
+ )
+ return tuple(policy for policy in admitted if counts[policy.model] == 1)
+
+
+#: Sentinel distinguishing "no index resolved yet" from an index resolved for
+#: no provider, which is the ordinary case.
+_UNRESOLVED: Any = object()
+
+
+@dataclass
+class _ResolvedRegistry:
+ """The index built for one provider, with the roots validated so far.
+
+ Matched to its provider by *identity* rather than keyed in an
+ ``lru_cache``: a cache key would have to be hashed, and a host callable
+ implemented as a mutable dataclass instance is unhashable. That raises
+ before the host boundary can isolate the host's mistake, which would
+ stop the built-in roots purging too.
+ """
+
+ provider: Any = _UNRESOLVED
+ registry: Mapping[type[Any], PurgeEntityPolicy] = field(
+ default_factory=lambda: MappingProxyType({})
+ )
+ validated: set[type[Any]] = field(default_factory=set)
+ #: Set only on a snapshot published because the provider could not be
+ #: called. Until this deadline the snapshot answers reads; after it, the
+ #: provider is tried again.
+ retry_after: float | None = None
+
+
+#: Key the resolved index lives under on the Flask app. Per app rather than
+#: per process: two apps in one process have different configs, and a single
+#: global snapshot would thrash between them -- re-resolving, and re-invoking
+#: the provider, on every read.
+_REGISTRY_EXTENSION_KEY: str = "deletion_retention_purge_registry"
+
+#: How long a snapshot published after a provider failure answers reads
+#: before the provider is tried again. Bounds both extremes: a failure is
+#: neither remembered for the life of the process nor retried on every read --
+#: at roughly three reads per purged entity, retrying per read meant tens of
+#: thousands of provider calls and tracebacks in a single pass.
+_PROVIDER_RETRY_SECONDS: float = 60.0
+
+#: Per-thread marker for "this thread is already resolving". A host provider
+#: may build its policy from a built-in one -- ``replace(get_purge_policy(
+#: Slice), model=HostRoot, ...)`` is the obvious way to borrow the stock
+#: callbacks -- which re-enters resolution while the first call is still in
+#: the provider. The nested call is handed the state already published
+#: instead, which always carries the built-in roots.
+_RESOLVING: threading.local = threading.local()
+
+
+def _resolved_registry() -> _ResolvedRegistry:
+ """Return the state resolved for the installed provider, building it once.
+
+ Readers take a snapshot and a rebuild publishes a new one, so a caller
+ always sees an index and its validation record together. Publishing is a
+ single rebind, which is what makes it safe to do this without a lock: two
+ concurrent first uses may each invoke the provider and each publish, but
+ the two snapshots are equivalent and neither is ever seen half-built.
+ Paying for a duplicate provider call is the deliberate trade -- a lock
+ here deadlocks the moment a provider resolves a built-in policy of its
+ own, and a hung purge is worse than a repeated call.
+ """
+ if not has_app_context():
+ return _builtin_registry()
+ provider: Callable[[], Any] | None = current_app.config.get(
+ HOST_POLICIES_CONFIG_KEY
+ )
+ published: _ResolvedRegistry = current_app.extensions.setdefault(
+ _REGISTRY_EXTENSION_KEY, _builtin_registry()
+ )
+ if published.provider is provider and (
+ published.retry_after is None or time.monotonic() < published.retry_after
+ ):
+ return published
+ if getattr(_RESOLVING, "active", False):
Review Comment:
The re-entrancy guard returns the built-in registry whenever a thread is resolving. A host policy that calls `get_purge_policy` for its own model inside the provider gets `Unsupported purge model`. Also, `_RESOLVING.active` is reset to False rather than restored, so it is only safe at one nesting level.
##########
superset/commands/deletion_retention/purge_policy.py:
##########
@@ -860,16 +914,396 @@ def declare(
cleanup_permission=cleanup_dataset_permission,
),
}
- return validate_unique_root_policies(registry.values())
+ return tuple(registry.values())
+
+
+def _policy_label(policy: Any) -> str:
+ """A log-safe name for a payload whose shape is not yet established."""
+ model: Any = getattr(policy, "model", None)
+ return str(getattr(model, "__name__", type(model).__name__))
+
+
+def _host_policy_payload(provided: Any) -> tuple[Any, ...]:
+ """Materialize the provider's payload, or reject its shape.
+
+ Any iterable is accepted -- a list, a tuple, ``dict.values()``, a
+ generator -- since the documented contract is a sequence of policies, not
+ one particular container. Strings and bytes are excluded because they
+ iterate into characters, which would read as a sequence of bad members
+ rather than the wrong type. Materialized once: a generator cannot be
+ walked twice.
+
+ An exception raised *while reading* a lazy payload is left to propagate.
+ Walking a generator runs host code just as calling the provider does, so
+ the two are the same kind of failure and earn the same retry; only the
+ shape of a payload that arrived intact is settled for good.
+ """
+ if isinstance(provided, (str, bytes)) or not isinstance(provided, Iterable):
+ logger.error(
+ "purge_policy: %s returned %s; expected an iterable of PurgeEntityPolicy",
+ HOST_POLICIES_CONFIG_KEY,
+ type(provided).__name__,
+ )
+ return ()
+ return tuple(provided)
+
+def _unqualified_table_conflict(policy: PurgeEntityPolicy) -> str | None:
+ """Return the first table a bare-name lookup would resolve wrongly.
-@lru_cache(maxsize=None)
-def _validated_purge_policy(model: type[Any]) -> PurgeEntityPolicy:
- """Validate and return one root policy without blocking unrelated roots."""
+ Discovery records table identities by bare name while a ``MetaData`` keys
+ them by schema, so a declared ``child`` resolves to the default-schema
+ table even where the policy meant ``host.child`` -- and the cleanup would
+ delete rows belonging to unrelated roots. Carrying schema-qualified
+ identities through discovery and execution is the better fix; until then
+ the ambiguous shape is refused rather than silently mis-resolved.
+ """
+ root_table: sa.Table = sa.inspect(policy.model).local_table
+ metadata: sa.MetaData = root_table.metadata
+ keys_by_name: dict[str, list[str]] = {}
+ for table in metadata.tables.values():
+ keys_by_name.setdefault(table.name, []).append(table.key)
+ declared: set[str] = {root_table.name} | {
+ dependency.key.related_table
+ for dependency in policy.dependencies
+ # Version shadows are resolved by bare name as well, in
+ # _entity_version_targets.
+ if dependency.classification in _EXECUTABLE_CLASSIFICATIONS
+ or dependency.classification is DependencyClassification.VERSION_OWNED
+ }
+ for name in sorted(declared):
+ keys: list[str] = keys_by_name.get(name, [])
+ if len(keys) != 1 or keys[0] != name:
+ return name
+ return None
+
+
+def _admitted_host_policy(
+ candidate: Any, builtin_roots: frozenset[type[Any]], builtin_types: frozenset[str]
+) -> PurgeEntityPolicy | None:
+ """Return *candidate* if it is a usable host policy, else ``None``.
+
+ Every check runs behind one handler rather than guarding each attribute:
+ the payload is host-supplied, so reading ``model`` or hashing it may
+ itself raise -- and an exception escaping here would break
+ ``purge_policy_registry`` for the built-in roots too, which is the
+ opposite of what this boundary is for.
+ """
try:
- policy: PurgeEntityPolicy = purge_policy_registry()[model]
- except KeyError as ex:
- raise ValueError(f"Unsupported purge model: {model.__name__}") from ex
+ if not isinstance(candidate, PurgeEntityPolicy):
+ logger.error(
+ "purge_policy: %s returned a %s; expected PurgeEntityPolicy",
+ HOST_POLICIES_CONFIG_KEY,
+ type(candidate).__name__,
+ )
+ return None
+ if not isinstance(candidate.model, type):
+ # Everything downstream treats the root as a mapped class:
+ # sa.inspect, the registry index, the duplicate count.
+ logger.error(
+ "purge_policy: host policy root %s is not a class",
+ _policy_label(candidate),
+ )
+ return None
+ if candidate.model in builtin_roots:
+ logger.error(
+ "purge_policy: host policy for built-in root %s ignored",
+ candidate.model.__name__,
+ )
+ return None
+ if candidate.entity_type in builtin_types:
+ # Core branches on entity_type -- the dataset impact path, the tag
+ # object type, the dataset permission name -- so a host reusing
+ # one of its names would have that logic pointed at its own rows.
+ logger.error(
+ "purge_policy: host policy for %s claims the reserved entity type %r",
+ candidate.model.__name__,
+ candidate.entity_type,
+ )
+ return None
+ # Validated first, so a table the metadata simply does not contain is
+ # reported as such rather than as an ambiguous name.
+ unusable: set[str] = {
+ dependency.classification.value
+ for dependency in candidate.dependencies
+ if dependency.classification
+ in {
+ DependencyClassification.BLOCK,
+ DependencyClassification.LISTENER_EFFECT,
+ }
+ }
+ if unusable:
+ # Two different reasons, both about identities this package owns.
+ # A stock listener action decides what to delete from a built-in
+ # entity type, so declared by a host it is accepted and then never
+ # runs. A blocker *would* be applied -- validate_deletion_allowed
+ # is generic over the declarations -- but its reason code lands in
+ # the purge audit record, whose vocabulary is a closed set pinned
+ # by test. A host refuses a purge from its own validator instead,
+ # where the code it records is plainly its own.
+ raise RuntimeError(
+ f"{', '.join(sorted(unusable))} dependencies are not available "
+ "to a host root"
+ )
+ _validated_policy(candidate)
+ conflict: str | None = _unqualified_table_conflict(candidate)
+ if conflict is not None:
+ raise RuntimeError(
+ f"table {conflict!r} cannot be identified unambiguously by "
+ "name; schema-qualified roots are not supported"
+ )
+ except Exception as ex: # pylint: disable=broad-except
+ # Validated at admission rather than when a row is purged: a
+ # declaration the cleanup cannot execute would otherwise fail once per
+ # eligible row, counted as a cascade failure, and stay invisible until
+ # something aged past the window.
+ logger.error(
+ "purge_policy: host policy for %s rejected: %s",
+ _policy_label(candidate),
+ ex,
+ )
+ return None
+ return candidate
+
+
+def _host_purge_policies(
+ provider: Callable[[], Any] | None,
+) -> tuple[PurgeEntityPolicy, ...] | None:
+ """Return the purge policies a host installed for its own roots.
+
+ A host distribution can carry ``SoftDeleteMixin`` entities this package
+ cannot import. The retention task discovers those entities through the
+ mixin registry, so without a policy they reach the cascade as an
+ unsupported model. The host therefore declares their purge behavior and
+ installs it under ``PURGE_POLICIES_FUNC``.
+
+ Host boundary: everything about a host policy is settled here, where it is
+ admitted -- an unavailable provider, a payload that is not an iterable of
+ policies, a root that is not a class, a collision with a root or entity
+ type declared in this package, two declarations for one root, and a
+ declaration the shared cleanup could not execute. Each is logged and
+ dropped, so that root is reported as unsupported instead of failing row by
+ row in a scheduled run. A broken host declaration must not take the purge
+ down with it, and must never redefine how a chart, dashboard or dataset is
+ purged.
+
+ Returns ``None`` for the failures that may be transient -- the provider
+ could not be called, or a lazy payload raised while being read. The caller
+ then publishes the built-in roots with a short deadline, so the next
+ resolution after it tries again rather than treating one bad moment as the
+ answer for the life of the process. A payload that arrived intact but is
+ the wrong shape is a host bug that will not fix itself, so it is published
+ as an empty result instead.
+ """
+ if provider is None:
+ return ()
+ try:
+ # Both the call and reading what it returns run host code, so a lazy
+ # payload that raises mid-walk is the same kind of failure as a
+ # provider that raises outright, and earns the same retry.
+ candidates: tuple[Any, ...] = _host_policy_payload(provider())
+ except Exception: # pylint: disable=broad-except
+ logger.exception(
+ "purge_policy: %s is unavailable; keeping built-in roots only",
+ HOST_POLICIES_CONFIG_KEY,
+ )
+ return None
+
+ builtin_policies: tuple[PurgeEntityPolicy, ...] = _builtin_purge_policies()
+ builtin_roots: frozenset[type[Any]] = frozenset(
+ policy.model for policy in builtin_policies
+ )
+ builtin_types: frozenset[str] = frozenset(
+ policy.entity_type for policy in builtin_policies
+ )
+ admitted: list[PurgeEntityPolicy] = [
+ policy
+ for candidate in candidates
+ if (policy := _admitted_host_policy(candidate, builtin_roots, builtin_types))
+ is not None
+ ]
+
+ # Counted after admission, where every root is known to be a class and so
+ # hashable. Which of two declarations for one root is authoritative is
+ # undecidable, and the loser would still delete rows, so both are dropped:
+ # the root is reported as unsupported, which is the recoverable outcome.
+ # It also keeps the duplicate away from validate_unique_root_policies,
+ # whose ValueError would abort the whole scheduled run.
+ counts: dict[type[Any], int] = {}
+ for policy in admitted:
+ counts[policy.model] = counts.get(policy.model, 0) + 1
+ for model, count in counts.items():
+ if count > 1:
+ logger.error(
+ "purge_policy: %s declares %d policies for %s; ignoring all of them",
+ HOST_POLICIES_CONFIG_KEY,
+ count,
+ model.__name__,
+ )
+ return tuple(policy for policy in admitted if counts[policy.model] == 1)
+
+
+#: Sentinel distinguishing "no index resolved yet" from an index resolved for
+#: no provider, which is the ordinary case.
+_UNRESOLVED: Any = object()
+
+
+@dataclass
+class _ResolvedRegistry:
+ """The index built for one provider, with the roots validated so far.
+
+ Matched to its provider by *identity* rather than keyed in an
+ ``lru_cache``: a cache key would have to be hashed, and a host callable
+ implemented as a mutable dataclass instance is unhashable. That raises
+ before the host boundary can isolate the host's mistake, which would
+ stop the built-in roots purging too.
+ """
+
+ provider: Any = _UNRESOLVED
+ registry: Mapping[type[Any], PurgeEntityPolicy] = field(
+ default_factory=lambda: MappingProxyType({})
+ )
+ validated: set[type[Any]] = field(default_factory=set)
+ #: Set only on a snapshot published because the provider could not be
+ #: called. Until this deadline the snapshot answers reads; after it, the
+ #: provider is tried again.
+ retry_after: float | None = None
+
+
+#: Key the resolved index lives under on the Flask app. Per app rather than
+#: per process: two apps in one process have different configs, and a single
+#: global snapshot would thrash between them -- re-resolving, and re-invoking
+#: the provider, on every read.
+_REGISTRY_EXTENSION_KEY: str = "deletion_retention_purge_registry"
+
+#: How long a snapshot published after a provider failure answers reads
+#: before the provider is tried again. Bounds both extremes: a failure is
+#: neither remembered for the life of the process nor retried on every read --
+#: at roughly three reads per purged entity, retrying per read meant tens of
+#: thousands of provider calls and tracebacks in a single pass.
+_PROVIDER_RETRY_SECONDS: float = 60.0
+
+#: Per-thread marker for "this thread is already resolving". A host provider
+#: may build its policy from a built-in one -- ``replace(get_purge_policy(
+#: Slice), model=HostRoot, ...)`` is the obvious way to borrow the stock
+#: callbacks -- which re-enters resolution while the first call is still in
+#: the provider. The nested call is handed the state already published
+#: instead, which always carries the built-in roots.
+_RESOLVING: threading.local = threading.local()
+
+
+def _resolved_registry() -> _ResolvedRegistry:
+ """Return the state resolved for the installed provider, building it once.
+
+ Readers take a snapshot and a rebuild publishes a new one, so a caller
+ always sees an index and its validation record together. Publishing is a
+ single rebind, which is what makes it safe to do this without a lock: two
+ concurrent first uses may each invoke the provider and each publish, but
+ the two snapshots are equivalent and neither is ever seen half-built.
+ Paying for a duplicate provider call is the deliberate trade -- a lock
+ here deadlocks the moment a provider resolves a built-in policy of its
+ own, and a hung purge is worse than a repeated call.
+ """
+ if not has_app_context():
+ return _builtin_registry()
+ provider: Callable[[], Any] | None = current_app.config.get(
+ HOST_POLICIES_CONFIG_KEY
+ )
+ published: _ResolvedRegistry = current_app.extensions.setdefault(
+ _REGISTRY_EXTENSION_KEY, _builtin_registry()
+ )
+ if published.provider is provider and (
Review Comment:
Registry state is read and published on the shared app extensions without a lock, and `validated` is a mutable set shared across threads. Concurrent first use can run the provider twice, and `current_app.extensions.setdefault(..., _builtin_registry())` evaluates eagerly on every call.
##########
superset/commands/deletion_retention/purge_policy.py:
##########
@@ -905,32 +1341,285 @@ def _validated_purge_policy(model: type[Any]) -> PurgeEntityPolicy:
details = (
f"{details}; " if details else ""
) + f"stale_listeners=[{', '.join(coverage.stale_listeners)}]"
- raise RuntimeError(f"Incomplete purge policy for {model.__name__}: {details}")
+ raise RuntimeError(
+ f"Incomplete purge policy for {policy.model.__name__}: {details}"
+ )
return policy
-def _validate_executable_declarations(policy: PurgeEntityPolicy) -> None:
- """Reject executable classifications missing their required action metadata."""
+def _ownership_dependency(
+ policy: PurgeEntityPolicy, related_table: str
+) -> DependencyPolicy | None:
+ """The single owned/association edge attaching *related_table*, if clear."""
+ candidates: tuple[DependencyPolicy, ...] = tuple(
+ dependency
+ for dependency in policy.dependencies
+ if dependency.classification
+ in {DependencyClassification.OWNED, DependencyClassification.ASSOCIATION}
+ and dependency.key.related_table == related_table
+ and dependency.key.direction == "inbound"
+ )
+ return candidates[0] if len(candidates) == 1 else None
+
+
+def _validate_owned_traversal(policy: PurgeEntityPolicy) -> None:
+ """Reject an owned table reachable only through an association.
+
+ The shared cleanup empties associations before owned children, while an
+ owned table's predicate selects its rows *through* its ownership path. If
+ a hop on that path is an association, its rows are already gone when the
+ owned delete runs, so the statement matches nothing.
+
+ Kept where the rest of the ordering rules were dropped because this one
+ can be silent: where foreign keys are enforced the association delete
+ fails loudly, but where they are not -- SQLite -- the root is purged and
+ its descendants are left orphaned with nothing reported. The remedy is to
+ classify the intermediate table as owned, which places it in the same
+ phase as what it leads to.
+ """
+ root_table: str = sa.inspect(policy.model).local_table.name
for dependency in policy.dependencies:
- if (
- dependency.classification is DependencyClassification.LISTENER_EFFECT
- and dependency.listener_action is None
- ):
- raise RuntimeError(
- f"Missing listener action for {dependency.key.describe()}"
- )
- if (
- dependency.classification is DependencyClassification.VERSION_OWNED
- and dependency.version_column is None
+ if dependency.classification is not DependencyClassification.OWNED:
+ continue
+ table_name: str = dependency.key.owner_table
+ visited: set[str] = set()
+ while table_name != root_table and table_name not in visited:
+ visited.add(table_name)
+ hop: DependencyPolicy | None = _ownership_dependency(policy, table_name)
+ if hop is None:
+ # An absent or ambiguous path is reported by coverage, and by
+ # _ownership_edge at execution; not this check's business.
+ break
+ if hop.classification is DependencyClassification.ASSOCIATION:
+ raise RuntimeError(
+ f"Owned dependency {dependency.key.describe()} is reachable "
+ f"only through association {hop.key.describe()}; "
+ "associations are deleted first, so the owned rows would "
+ "be orphaned"
+ )
+ table_name = hop.key.owner_table
+
+
+def _validate_recursive_ownership(policy: PurgeEntityPolicy) -> None:
+ """Reject a self-referencing owned table under the stock cleanup.
+
+ ``delete_owned_children`` issues one statement per declared edge, so a
+ table that owns itself is pruned one level deep. Where foreign keys are
+ enforced the root's own delete then fails and rolls back; where they are
+ not -- SQLite -- the root is purged and its grandchildren are left behind
+ with a dangling parent id and nothing reported. A host declaring a tree
+ supplies cleanup that walks it.
+ """
+ for dependency in policy.dependencies:
+ if dependency.classification is not DependencyClassification.OWNED:
+ continue
+ key: DependencyKey = dependency.key
+ if key.owner_table != key.related_table:
+ continue
+ raise RuntimeError(
+ f"Owned dependency {key.describe()} is self-referencing; the stock "
+ "owned-child cleanup deletes one level, so this policy must supply "
+ "its own delete_owned_children"
+ )
+
+
+def _validate_scanner_requirements(policy: PurgeEntityPolicy) -> None:
+ """Reject a root the scheduled scan cannot page through.
+
+ The retention task selects, windows and orders eligible rows by ``id``,
+ and the cascade pins each row by it. A root keyed on something else -- a
+ UUID primary key with no ``id`` column -- raises inside the scan, outside
+ the per-row error handling, so the run aborts before the remaining roots
+ are reached.
+ """
+ mapper: Mapper[Any] = sa.inspect(policy.model)
+ table: sa.Table = mapper.local_table
+ if "id" not in table.c:
+ raise RuntimeError(
+ f"Purge root {policy.model.__name__} has no 'id' column; the "
+ "scheduled scan pages eligible rows by id"
+ )
+ primary_key: tuple[sa.Column[Any], ...] = tuple(mapper.primary_key)
+ if (
+ len(primary_key) != 1
+ or primary_key[0].name != "id"
+ or not isinstance(primary_key[0].type, sa.Integer)
+ ):
+ # The scan pages by id and the row is then fetched with a scalar
+ # Session.get, so a composite or differently named key matches
+ # nothing: every row fails while a dry run still counts it eligible.
+ raise RuntimeError(
+ f"Purge root {policy.model.__name__} is keyed on "
+ f"({', '.join(column.name for column in primary_key)}); the "
+ "scheduled purge pages an integer 'id' watermark and looks rows "
+ "up by it"
+ )
+ if "deleted_at" not in table.c:
+ raise RuntimeError(
+ f"Purge root {policy.model.__name__} has no 'deleted_at' column; "
+ "every purge path requires the row to be archived first"
+ )
+
+
+#: Classifications the shared cleanup executes as SQL, through
+#: ``_dependency_predicates`` -- which only knows how to read an inbound
+#: foreign key.
+_EXECUTABLE_CLASSIFICATIONS: frozenset[DependencyClassification] = frozenset(
+ {
+ DependencyClassification.OWNED,
+ DependencyClassification.ASSOCIATION,
+ DependencyClassification.BLOCK,
+ }
+)
+
+
+def _validate_versioned_root(policy: PurgeEntityPolicy) -> None:
+ """Reject version targets on a root that has no version class.
+
+ ``_delete_version_history`` resolves the *root's* version class first and
+ returns as soon as that raises, so a target declared on a child's shadow
+ is never reached for an unversioned root: the root and its live child rows
+ are purged while the child's history survives, with nothing reported.
+
+ Skipped where nothing is versioned at all. The version cascade is then
+ inert for every root, including this package's own, and refusing would
+ only take those down with it.
+ """
+ try:
+ from sqlalchemy_continuum import version_class, versioning_manager
+ from sqlalchemy_continuum.exc import ClassNotVersioned
+ except ImportError: # pragma: no cover - versioning not installed
+ return
+ if not getattr(versioning_manager, "version_class_map", None):
Review Comment:
When `version_class_map` is empty, the unversioned-root check is skipped. Versioning set up lazily after admission would then leave the check unenforced for good, because validated roots are never re-checked.
##########
superset/commands/deletion_retention/purge_policy.py:
##########
@@ -860,16 +914,396 @@ def declare(
cleanup_permission=cleanup_dataset_permission,
),
}
- return validate_unique_root_policies(registry.values())
+ return tuple(registry.values())
+
+
+def _policy_label(policy: Any) -> str:
+ """A log-safe name for a payload whose shape is not yet established."""
+ model: Any = getattr(policy, "model", None)
+ return str(getattr(model, "__name__", type(model).__name__))
+
+
+def _host_policy_payload(provided: Any) -> tuple[Any, ...]:
+ """Materialize the provider's payload, or reject its shape.
+
+ Any iterable is accepted -- a list, a tuple, ``dict.values()``, a
+ generator -- since the documented contract is a sequence of policies, not
+ one particular container. Strings and bytes are excluded because they
+ iterate into characters, which would read as a sequence of bad members
+ rather than the wrong type. Materialized once: a generator cannot be
+ walked twice.
+
+ An exception raised *while reading* a lazy payload is left to propagate.
+ Walking a generator runs host code just as calling the provider does, so
+ the two are the same kind of failure and earn the same retry; only the
+ shape of a payload that arrived intact is settled for good.
+ """
+ if isinstance(provided, (str, bytes)) or not isinstance(provided, Iterable):
+ logger.error(
+ "purge_policy: %s returned %s; expected an iterable of PurgeEntityPolicy",
+ HOST_POLICIES_CONFIG_KEY,
+ type(provided).__name__,
+ )
+ return ()
+ return tuple(provided)
+
+def _unqualified_table_conflict(policy: PurgeEntityPolicy) -> str | None:
+ """Return the first table a bare-name lookup would resolve wrongly.
-@lru_cache(maxsize=None)
-def _validated_purge_policy(model: type[Any]) -> PurgeEntityPolicy:
- """Validate and return one root policy without blocking unrelated roots."""
+ Discovery records table identities by bare name while a ``MetaData`` keys
+ them by schema, so a declared ``child`` resolves to the default-schema
+ table even where the policy meant ``host.child`` -- and the cleanup would
+ delete rows belonging to unrelated roots. Carrying schema-qualified
+ identities through discovery and execution is the better fix; until then
+ the ambiguous shape is refused rather than silently mis-resolved.
+ """
+ root_table: sa.Table = sa.inspect(policy.model).local_table
+ metadata: sa.MetaData = root_table.metadata
+ keys_by_name: dict[str, list[str]] = {}
+ for table in metadata.tables.values():
+ keys_by_name.setdefault(table.name, []).append(table.key)
+ declared: set[str] = {root_table.name} | {
+ dependency.key.related_table
+ for dependency in policy.dependencies
+ # Version shadows are resolved by bare name as well, in
+ # _entity_version_targets.
+ if dependency.classification in _EXECUTABLE_CLASSIFICATIONS
+ or dependency.classification is DependencyClassification.VERSION_OWNED
+ }
+ for name in sorted(declared):
+ keys: list[str] = keys_by_name.get(name, [])
+ if len(keys) != 1 or keys[0] != name:
+ return name
+ return None
+
+
+def _admitted_host_policy(
+ candidate: Any, builtin_roots: frozenset[type[Any]], builtin_types: frozenset[str]
+) -> PurgeEntityPolicy | None:
+ """Return *candidate* if it is a usable host policy, else ``None``.
+
+ Every check runs behind one handler rather than guarding each attribute:
+ the payload is host-supplied, so reading ``model`` or hashing it may
+ itself raise -- and an exception escaping here would break
+ ``purge_policy_registry`` for the built-in roots too, which is the
+ opposite of what this boundary is for.
+ """
try:
- policy: PurgeEntityPolicy = purge_policy_registry()[model]
- except KeyError as ex:
- raise ValueError(f"Unsupported purge model: {model.__name__}") from ex
+ if not isinstance(candidate, PurgeEntityPolicy):
+ logger.error(
+ "purge_policy: %s returned a %s; expected PurgeEntityPolicy",
+ HOST_POLICIES_CONFIG_KEY,
+ type(candidate).__name__,
+ )
+ return None
+ if not isinstance(candidate.model, type):
+ # Everything downstream treats the root as a mapped class:
+ # sa.inspect, the registry index, the duplicate count.
+ logger.error(
+ "purge_policy: host policy root %s is not a class",
+ _policy_label(candidate),
+ )
+ return None
+ if candidate.model in builtin_roots:
Review Comment:
The built-in root check compares by model class only. A host policy with a different class mapped onto a built-in table (e.g. a subclass of Slice) passes admission and gets a host cleanup path against core tables.
##########
superset/commands/deletion_retention/purge_policy.py:
##########
@@ -905,32 +1341,285 @@ def _validated_purge_policy(model: type[Any]) -> PurgeEntityPolicy:
details = (
f"{details}; " if details else ""
) + f"stale_listeners=[{', '.join(coverage.stale_listeners)}]"
- raise RuntimeError(f"Incomplete purge policy for {model.__name__}: {details}")
+ raise RuntimeError(
+ f"Incomplete purge policy for {policy.model.__name__}: {details}"
+ )
return policy
-def _validate_executable_declarations(policy: PurgeEntityPolicy) -> None:
- """Reject executable classifications missing their required action metadata."""
+def _ownership_dependency(
+ policy: PurgeEntityPolicy, related_table: str
+) -> DependencyPolicy | None:
+ """The single owned/association edge attaching *related_table*, if clear."""
+ candidates: tuple[DependencyPolicy, ...] = tuple(
+ dependency
+ for dependency in policy.dependencies
+ if dependency.classification
+ in {DependencyClassification.OWNED, DependencyClassification.ASSOCIATION}
+ and dependency.key.related_table == related_table
+ and dependency.key.direction == "inbound"
+ )
+ return candidates[0] if len(candidates) == 1 else None
+
+
+def _validate_owned_traversal(policy: PurgeEntityPolicy) -> None:
+ """Reject an owned table reachable only through an association.
+
+ The shared cleanup empties associations before owned children, while an
+ owned table's predicate selects its rows *through* its ownership path. If
+ a hop on that path is an association, its rows are already gone when the
+ owned delete runs, so the statement matches nothing.
+
+ Kept where the rest of the ordering rules were dropped because this one
+ can be silent: where foreign keys are enforced the association delete
+ fails loudly, but where they are not -- SQLite -- the root is purged and
+ its descendants are left orphaned with nothing reported. The remedy is to
+ classify the intermediate table as owned, which places it in the same
+ phase as what it leads to.
+ """
+ root_table: str = sa.inspect(policy.model).local_table.name
for dependency in policy.dependencies:
- if (
- dependency.classification is DependencyClassification.LISTENER_EFFECT
- and dependency.listener_action is None
- ):
- raise RuntimeError(
- f"Missing listener action for {dependency.key.describe()}"
- )
- if (
- dependency.classification is DependencyClassification.VERSION_OWNED
- and dependency.version_column is None
+ if dependency.classification is not DependencyClassification.OWNED:
+ continue
+ table_name: str = dependency.key.owner_table
+ visited: set[str] = set()
+ while table_name != root_table and table_name not in visited:
+ visited.add(table_name)
+ hop: DependencyPolicy | None = _ownership_dependency(policy, table_name)
+ if hop is None:
+ # An absent or ambiguous path is reported by coverage, and by
+ # _ownership_edge at execution; not this check's business.
+ break
+ if hop.classification is DependencyClassification.ASSOCIATION:
+ raise RuntimeError(
+ f"Owned dependency {dependency.key.describe()} is reachable "
+ f"only through association {hop.key.describe()}; "
+ "associations are deleted first, so the owned rows would "
+ "be orphaned"
+ )
+ table_name = hop.key.owner_table
+
+
+def _validate_recursive_ownership(policy: PurgeEntityPolicy) -> None:
+ """Reject a self-referencing owned table under the stock cleanup.
+
+ ``delete_owned_children`` issues one statement per declared edge, so a
+ table that owns itself is pruned one level deep. Where foreign keys are
+ enforced the root's own delete then fails and rolls back; where they are
+ not -- SQLite -- the root is purged and its grandchildren are left behind
+ with a dangling parent id and nothing reported. A host declaring a tree
+ supplies cleanup that walks it.
+ """
+ for dependency in policy.dependencies:
+ if dependency.classification is not DependencyClassification.OWNED:
+ continue
+ key: DependencyKey = dependency.key
+ if key.owner_table != key.related_table:
+ continue
+ raise RuntimeError(
+ f"Owned dependency {key.describe()} is self-referencing; the stock "
+ "owned-child cleanup deletes one level, so this policy must supply "
+ "its own delete_owned_children"
+ )
+
+
+def _validate_scanner_requirements(policy: PurgeEntityPolicy) -> None:
+ """Reject a root the scheduled scan cannot page through.
+
+ The retention task selects, windows and orders eligible rows by ``id``,
+ and the cascade pins each row by it. A root keyed on something else -- a
+ UUID primary key with no ``id`` column -- raises inside the scan, outside
+ the per-row error handling, so the run aborts before the remaining roots
+ are reached.
+ """
+ mapper: Mapper[Any] = sa.inspect(policy.model)
+ table: sa.Table = mapper.local_table
+ if "id" not in table.c:
+ raise RuntimeError(
+ f"Purge root {policy.model.__name__} has no 'id' column; the "
+ "scheduled scan pages eligible rows by id"
+ )
+ primary_key: tuple[sa.Column[Any], ...] = tuple(mapper.primary_key)
+ if (
+ len(primary_key) != 1
+ or primary_key[0].name != "id"
+ or not isinstance(primary_key[0].type, sa.Integer)
+ ):
+ # The scan pages by id and the row is then fetched with a scalar
+ # Session.get, so a composite or differently named key matches
+ # nothing: every row fails while a dry run still counts it eligible.
+ raise RuntimeError(
+ f"Purge root {policy.model.__name__} is keyed on "
+ f"({', '.join(column.name for column in primary_key)}); the "
+ "scheduled purge pages an integer 'id' watermark and looks rows "
+ "up by it"
+ )
+ if "deleted_at" not in table.c:
+ raise RuntimeError(
+ f"Purge root {policy.model.__name__} has no 'deleted_at' column; "
+ "every purge path requires the row to be archived first"
+ )
+
+
+#: Classifications the shared cleanup executes as SQL, through
+#: ``_dependency_predicates`` -- which only knows how to read an inbound
+#: foreign key.
+_EXECUTABLE_CLASSIFICATIONS: frozenset[DependencyClassification] = frozenset(
+ {
+ DependencyClassification.OWNED,
+ DependencyClassification.ASSOCIATION,
+ DependencyClassification.BLOCK,
+ }
+)
+
+
+def _validate_versioned_root(policy: PurgeEntityPolicy) -> None:
+ """Reject version targets on a root that has no version class.
+
+ ``_delete_version_history`` resolves the *root's* version class first and
+ returns as soon as that raises, so a target declared on a child's shadow
+ is never reached for an unversioned root: the root and its live child rows
+ are purged while the child's history survives, with nothing reported.
+
+ Skipped where nothing is versioned at all. The version cascade is then
+ inert for every root, including this package's own, and refusing would
+ only take those down with it.
+ """
+ try:
+ from sqlalchemy_continuum import version_class, versioning_manager
+ from sqlalchemy_continuum.exc import ClassNotVersioned
+ except ImportError: # pragma: no cover - versioning not installed
+ return
+ if not getattr(versioning_manager, "version_class_map", None):
+ return
+ try:
+ version_class(policy.model)
+ except ClassNotVersioned:
+ raise RuntimeError(
+ f"Purge root {policy.model.__name__} has no version class, so the "
+ "version cascade returns before reaching a declared target; such "
+ "a policy declares none"
+ ) from None
+
+
+def _validate_version_target(
+ policy: PurgeEntityPolicy, dependency: DependencyPolicy, metadata: sa.MetaData
+) -> None:
+ """Reject a version target whose column does not hold the root's id.
+
+ Version cleanup compares the declared column with the root's id directly,
+ so the column has to be root-relative. Shadow tables are generated and may
+ not exist when a policy is validated, so the check reads their live
+ counterpart -- ``<table>_version`` describes ``<table>``, the naming the
+ shadow declarations already rely on -- and accepts the column when it is
+ the root's own primary key or a foreign key into the root.
+
+ A column naming an intermediate owner, a stage id under a workflow root
+ say, matches history belonging to a different root and leaves this one's
+ behind.
+ """
+ root_table: sa.Table = sa.inspect(policy.model).local_table
+ column: str = cast(str, dependency.version_column)
+ shadow_name: str = dependency.key.related_table
+ suffix: str = "_version"
+ live_name: str = (
+ shadow_name[: -len(suffix)] if shadow_name.endswith(suffix) else shadow_name
+ )
+ live_table: sa.Table | None = metadata.tables.get(live_name)
+ if live_table is not None and column in live_table.c:
+ live_column: sa.Column[Any] = live_table.c[column]
+ primary_key: tuple[sa.Column[Any], ...] = tuple(root_table.primary_key.columns)
+ if live_table is root_table:
+ # On the root's own shadow only its key identifies it. A
+ # self-referencing column such as ``parent_id`` points at *other*
+ # rows of the same root, so cleanup would delete the children's
+ # history and leave the purged row's own behind.
+ if len(primary_key) == 1 and live_column.name == primary_key[0].name:
+ return
+ # Elsewhere the comparison is still against the root's id, so a
+ # foreign key to any other root column matches rows belonging to a
+ # different root.
+ elif len(primary_key) == 1 and any(
+ foreign_key.column.table is root_table
+ and foreign_key.column.name == primary_key[0].name
+ for foreign_key in live_column.foreign_keys
):
- raise RuntimeError(
- f"Missing version target column for {dependency.key.describe()}"
- )
+ return
+ raise RuntimeError(
+ f"Version target {shadow_name}.{column} for {dependency.key.describe()} "
+ f"is not root-relative; cleanup compares it with the "
+ f"{root_table.name} id, so it must be that id or a foreign key to it"
+ )
+
+
+def _validate_dependency_declaration(
+ policy: PurgeEntityPolicy,
+ dependency: DependencyPolicy,
+ metadata: sa.MetaData,
+) -> None:
+ """Reject one declaration whose mistake would not announce itself."""
+ key: DependencyKey = dependency.key
+ if (
+ dependency.classification is DependencyClassification.LISTENER_EFFECT
+ and dependency.listener_action is None
+ ):
+ raise RuntimeError(f"Missing listener action for {key.describe()}")
+ if dependency.classification is DependencyClassification.VERSION_OWNED:
+ if dependency.version_column is None:
+ raise RuntimeError(f"Missing version target column for {key.describe()}")
+ # Both silent either way: a target the version cascade never
+ # reaches, and one that is not root-relative and so deletes another
+ # root's history while leaving this root's behind.
+ _validate_versioned_root(policy)
+ _validate_version_target(policy, dependency, metadata)
+
+
+def _validate_executable_declarations(policy: PurgeEntityPolicy) -> None:
+ """Reject a declaration that would delete the wrong rows, and little else.
+
+ Validation here is deliberately not an attempt to prove that an arbitrary
+ declaration executes correctly -- the space of shapes a host might write
+ is not enumerable, and every rule written to anticipate one is a rule to
+ maintain. The line drawn instead:
+
+ * a declaration whose mistake would **delete the wrong rows, or silently
+ skip cleanup**, is refused, because neither announces itself;
+ * a declaration that merely **fails loudly** is left to fail. A predicate
+ the cleanup cannot build raises per row and is counted as a cascade
+ failure; an ordering a foreign key refuses surfaces as an integrity
+ error, which the cascade reports as a blocked purge carrying the
+ ``cascade_integrity_failure`` reason code. Either way the attempt is
+ recorded, isolated to its own root by the scheduled task, and the
+ entity is retried on the next run.
+
+ The frame's own requirements are checked regardless, since the scan and
+ the locked claim read them before any policy code runs.
+ """
+ metadata: sa.MetaData = sa.inspect(policy.model).local_table.metadata
+ for dependency in policy.dependencies:
+ _validate_dependency_declaration(policy, dependency, metadata)
+ _validate_scanner_requirements(policy)
+ if policy.delete_owned_children is delete_owned_children:
+ # The hazard is the stock owned cleanup's: it builds each predicate by
+ # traversing the ownership path, which is empty by then whoever
+ # deleted the association rows. A policy that walks its own tree is
+ # not exposed to it; one that merely replaces the association delete
+ # still is.
+ _validate_owned_traversal(policy)
+ _validate_recursive_ownership(policy)
def get_purge_policy(model: type[Any]) -> PurgeEntityPolicy:
"""Resolve a complete policy or reject an unsupported purge model."""
- return _validated_purge_policy(cast(Hashable, model))
+ resolved: _ResolvedRegistry = _resolved_registry()
+ try:
+ policy: PurgeEntityPolicy = resolved.registry[model]
+ except KeyError as ex:
+ raise ValueError(f"Unsupported purge model: {model.__name__}") from ex
+ if model not in resolved.validated:
+ _validated_policy(policy)
Review Comment:
`get_purge_policy` validates built-ins lazily and mutates `resolved.validated` after the check. When the validation raises, it is re-run on every call, which could be once per row, and the traceback repeats each time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(retention): let a host install purge policies for its own soft-delete roots [superset]
Posted by "sadpandajoe (via GitHub)" <gi...@apache.org>.
sadpandajoe commented on code in PR #44892:
URL: https://github.com/apache/superset/pull/44892#discussion_r4172315726
##########
superset/commands/deletion_retention/purge_policy.py:
##########
@@ -860,7 +867,96 @@ def declare(
cleanup_permission=cleanup_dataset_permission,
),
}
- return validate_unique_root_policies(registry.values())
+ return tuple(registry.values())
+
+
+def _host_purge_policies() -> tuple[PurgeEntityPolicy, ...]:
+ """Return the purge policies a host installed for its own roots.
+
+ A host distribution can carry ``SoftDeleteMixin`` entities this package
+ cannot import. The retention task discovers those entities through the
+ mixin registry, so without a policy they reach the cascade as an
+ unsupported model. The host therefore declares their purge behavior and
+ installs it under ``PURGE_POLICIES_FUNC``.
+
+ Host boundary: an unavailable provider, a malformed payload, a policy that
+ collides with a root declared here, and two policies for one host root are
+ each logged and dropped. A broken host declaration must not take the
+ scheduled purge down with it, and must never redefine how a chart,
+ dashboard or dataset is purged.
+ """
+ if not has_app_context():
+ return ()
+ provider: Callable[[], Any] | None = current_app.config.get(
+ HOST_POLICIES_CONFIG_KEY
+ )
+ if provider is None:
+ return ()
+ try:
+ provided: Any = provider()
+ except Exception: # pylint: disable=broad-except
+ logger.exception(
+ "purge_policy: %s is unavailable; keeping built-in roots only",
+ HOST_POLICIES_CONFIG_KEY,
+ )
+ return ()
+ if not isinstance(provided, (list, tuple)) or not all(
+ isinstance(policy, PurgeEntityPolicy) for policy in provided
+ ):
+ logger.error(
+ "purge_policy: %s returned %s; expected a sequence of PurgeEntityPolicy",
+ HOST_POLICIES_CONFIG_KEY,
+ type(provided).__name__,
+ )
+ return ()
+ builtin_roots: frozenset[type[Any]] = frozenset(
+ policy.model for policy in _builtin_purge_policies()
+ )
+ declared: dict[type[Any], int] = {}
+ for policy in provided:
+ declared[policy.model] = declared.get(policy.model, 0) + 1
+ # Which of two declarations for one root is authoritative is undecidable,
+ # and the loser would still delete rows. Dropping both leaves the model
+ # reported as unsupported, which is the recoverable outcome. Resolving it
+ # here also keeps the duplicate away from validate_unique_root_policies,
+ # whose ValueError would abort the whole scheduled run.
+ duplicated: list[type[Any]] = [
+ model for model, count in declared.items() if count > 1
+ ]
+ for model in duplicated:
+ logger.error(
+ "purge_policy: %s declares %d policies for %s; ignoring all of them",
+ HOST_POLICIES_CONFIG_KEY,
+ declared[model],
+ model.__name__,
+ )
+ accepted: list[PurgeEntityPolicy] = []
+ for policy in provided:
+ if policy.model in builtin_roots:
+ logger.error(
+ "purge_policy: host policy for built-in root %s ignored",
+ policy.model.__name__,
+ )
+ continue
+ if policy.model in duplicated:
+ continue
+ accepted.append(policy)
+ return tuple(accepted)
+
+
+def purge_policy_registry() -> Mapping[type[Any], PurgeEntityPolicy]:
+ """Index the built-in purge roots plus any the host installed.
+
+ Uncached on purpose, unlike its two inputs: the built-in declarations are
+ built once per process, while a host policy is resolved per call, so a
+ provider installed after the first purge is honored for any root not yet
+ resolved. ``get_purge_policy`` memoizes per model, so replacing the policy
+ of a root it has already resolved needs a restart -- and that memoization
+ is why rebuilding this small index is off the hot path.
+ """
+ return validate_unique_root_policies(
Review Comment:
The new hook admits larger host graphs into discovery, which revisits every remaining declared table for each traversal permutation: eight directly owned tables take 109,600 visits for 16 unique dependency keys, and twelve require about 1.3 billion visits before any row can purge. Could discovery track visited tables across the whole graph before accepting these policies?
##########
superset/commands/deletion_retention/purge_policy.py:
##########
@@ -860,7 +867,96 @@ def declare(
cleanup_permission=cleanup_dataset_permission,
),
}
- return validate_unique_root_policies(registry.values())
+ return tuple(registry.values())
+
+
+def _host_purge_policies() -> tuple[PurgeEntityPolicy, ...]:
+ """Return the purge policies a host installed for its own roots.
+
+ A host distribution can carry ``SoftDeleteMixin`` entities this package
+ cannot import. The retention task discovers those entities through the
+ mixin registry, so without a policy they reach the cascade as an
+ unsupported model. The host therefore declares their purge behavior and
+ installs it under ``PURGE_POLICIES_FUNC``.
+
+ Host boundary: an unavailable provider, a malformed payload, a policy that
+ collides with a root declared here, and two policies for one host root are
+ each logged and dropped. A broken host declaration must not take the
+ scheduled purge down with it, and must never redefine how a chart,
+ dashboard or dataset is purged.
+ """
+ if not has_app_context():
+ return ()
+ provider: Callable[[], Any] | None = current_app.config.get(
+ HOST_POLICIES_CONFIG_KEY
+ )
+ if provider is None:
+ return ()
+ try:
+ provided: Any = provider()
+ except Exception: # pylint: disable=broad-except
+ logger.exception(
+ "purge_policy: %s is unavailable; keeping built-in roots only",
+ HOST_POLICIES_CONFIG_KEY,
+ )
+ return ()
+ if not isinstance(provided, (list, tuple)) or not all(
+ isinstance(policy, PurgeEntityPolicy) for policy in provided
+ ):
+ logger.error(
+ "purge_policy: %s returned %s; expected a sequence of PurgeEntityPolicy",
+ HOST_POLICIES_CONFIG_KEY,
+ type(provided).__name__,
+ )
+ return ()
+ builtin_roots: frozenset[type[Any]] = frozenset(
+ policy.model for policy in _builtin_purge_policies()
+ )
+ declared: dict[type[Any], int] = {}
+ for policy in provided:
+ declared[policy.model] = declared.get(policy.model, 0) + 1
+ # Which of two declarations for one root is authoritative is undecidable,
+ # and the loser would still delete rows. Dropping both leaves the model
+ # reported as unsupported, which is the recoverable outcome. Resolving it
+ # here also keeps the duplicate away from validate_unique_root_policies,
+ # whose ValueError would abort the whole scheduled run.
+ duplicated: list[type[Any]] = [
+ model for model, count in declared.items() if count > 1
+ ]
+ for model in duplicated:
+ logger.error(
+ "purge_policy: %s declares %d policies for %s; ignoring all of them",
+ HOST_POLICIES_CONFIG_KEY,
+ declared[model],
+ model.__name__,
+ )
+ accepted: list[PurgeEntityPolicy] = []
+ for policy in provided:
+ if policy.model in builtin_roots:
+ logger.error(
+ "purge_policy: host policy for built-in root %s ignored",
+ policy.model.__name__,
+ )
+ continue
+ if policy.model in duplicated:
+ continue
+ accepted.append(policy)
Review Comment:
A graph-complete host model with a UUID primary key and no `id` column is accepted here, but the scheduled scan raises at `table.c.id` outside its per-row error handling—even on an empty table or dry run—and skips any later built-in roots. Could host admission validate the stock scanner’s root requirements, or could scanning failures be isolated per model?
##########
superset/commands/deletion_retention/purge_policy.py:
##########
@@ -926,6 +1074,7 @@ def _validate_executable_declarations(policy: PurgeEntityPolicy) -> None:
raise RuntimeError(
f"Missing version target column for {dependency.key.describe()}"
)
+ _validate_owned_traversal(policy)
Review Comment:
A complete host policy owning both assets and tasks, with `task.asset_id → asset.id` preserved as a cross-link, passes this validation, but both deletes have the same depth: declaring assets first makes every purge roll back under enforced FKs, while reversing the declarations succeeds. Could validation reject unsupported sibling cross-links, or could cleanup order them by their foreign-key dependencies rather than declaration order?
##########
superset/commands/deletion_retention/purge_policy.py:
##########
@@ -909,6 +1005,58 @@ def _validated_purge_policy(model: type[Any]) -> PurgeEntityPolicy:
return policy
+def _ownership_dependency(
+ policy: PurgeEntityPolicy, related_table: str
+) -> DependencyPolicy | None:
+ """The single owned/association edge attaching *related_table*, if clear."""
+ candidates: tuple[DependencyPolicy, ...] = tuple(
+ dependency
+ for dependency in policy.dependencies
+ if dependency.classification
+ in {DependencyClassification.OWNED, DependencyClassification.ASSOCIATION}
+ and dependency.key.related_table == related_table
+ and dependency.key.direction == "inbound"
+ )
+ return candidates[0] if len(candidates) == 1 else None
+
+
+def _validate_owned_traversal(policy: PurgeEntityPolicy) -> None:
+ """Reject an owned table reachable only through an association.
+
+ ``cascade_hard_delete`` empties associations before owned children, while
+ an owned table's predicate selects its rows *through* its ownership path
+ (see ``_owner_value_select``). If a hop on that path is an association,
+ its rows are already gone when the owned delete runs: the statement
+ matches nothing, leaving the descendants orphaned where foreign keys are
+ unenforced and blocking the root's delete where they are not.
+
+ Refused at declaration time rather than executed. The shape has a
+ remedy -- classify the intermediate table as owned, which places it in
+ the same phase as what it leads to.
+ """
+ root_table: str = sa.inspect(policy.model).local_table.name
+ for dependency in policy.dependencies:
+ if dependency.classification is not DependencyClassification.OWNED:
+ continue
+ table_name: str = dependency.key.owner_table
+ visited: set[str] = set()
+ while table_name != root_table and table_name not in visited:
Review Comment:
An `OWNED` self-FK passes this check because its owner table is already the root, but a host tree `1 → 2 → 3` is only deleted one level deep: with SQLite FKs disabled, purge reports success after removing 1 and 2 while leaving 3 orphaned; restrictive FKs instead roll the purge back. Could recursive ownership be rejected here unless the cleanup actually deletes the full subtree?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(retention): let a host install purge policies for its own soft-delete roots [superset]
Posted by "alexandrusoare (via GitHub)" <gi...@apache.org>.
alexandrusoare commented on code in PR #44892:
URL: https://github.com/apache/superset/pull/44892#discussion_r4183592916
##########
superset/commands/deletion_retention/purge_policy.py:
##########
@@ -860,7 +894,96 @@ def declare(
cleanup_permission=cleanup_dataset_permission,
),
}
- return validate_unique_root_policies(registry.values())
+ return tuple(registry.values())
+
+
+def _host_purge_policies() -> tuple[PurgeEntityPolicy, ...]:
+ """Return the purge policies a host installed for its own roots.
+
+ A host distribution can carry ``SoftDeleteMixin`` entities this package
+ cannot import. The retention task discovers those entities through the
+ mixin registry, so without a policy they reach the cascade as an
+ unsupported model. The host therefore declares their purge behavior and
+ installs it under ``PURGE_POLICIES_FUNC``.
+
+ Host boundary: an unavailable provider, a malformed payload, a policy that
+ collides with a root declared here, and two policies for one host root are
+ each logged and dropped. A broken host declaration must not take the
+ scheduled purge down with it, and must never redefine how a chart,
+ dashboard or dataset is purged.
+ """
+ if not has_app_context():
+ return ()
+ provider: Callable[[], Any] | None = current_app.config.get(
+ HOST_POLICIES_CONFIG_KEY
+ )
+ if provider is None:
+ return ()
+ try:
+ provided: Any = provider()
+ except Exception: # pylint: disable=broad-except
+ logger.exception(
+ "purge_policy: %s is unavailable; keeping built-in roots only",
+ HOST_POLICIES_CONFIG_KEY,
+ )
+ return ()
+ if not isinstance(provided, (list, tuple)) or not all(
+ isinstance(policy, PurgeEntityPolicy) for policy in provided
+ ):
+ logger.error(
+ "purge_policy: %s returned %s; expected a sequence of PurgeEntityPolicy",
+ HOST_POLICIES_CONFIG_KEY,
+ type(provided).__name__,
+ )
+ return ()
+ builtin_roots: frozenset[type[Any]] = frozenset(
+ policy.model for policy in _builtin_purge_policies()
+ )
+ declared: dict[type[Any], int] = {}
+ for policy in provided:
+ declared[policy.model] = declared.get(policy.model, 0) + 1
+ # Which of two declarations for one root is authoritative is undecidable,
+ # and the loser would still delete rows. Dropping both leaves the model
+ # reported as unsupported, which is the recoverable outcome. Resolving it
+ # here also keeps the duplicate away from validate_unique_root_policies,
+ # whose ValueError would abort the whole scheduled run.
+ duplicated: list[type[Any]] = [
+ model for model, count in declared.items() if count > 1
+ ]
+ for model in duplicated:
+ logger.error(
+ "purge_policy: %s declares %d policies for %s; ignoring all of them",
+ HOST_POLICIES_CONFIG_KEY,
+ declared[model],
+ model.__name__,
+ )
+ accepted: list[PurgeEntityPolicy] = []
+ for policy in provided:
+ if policy.model in builtin_roots:
+ logger.error(
+ "purge_policy: host policy for built-in root %s ignored",
+ policy.model.__name__,
+ )
+ continue
+ if policy.model in duplicated:
+ continue
+ accepted.append(policy)
+ return tuple(accepted)
+
+
+def purge_policy_registry() -> Mapping[type[Any], PurgeEntityPolicy]:
+ """Index the built-in purge roots plus any the host installed.
+
+ Uncached on purpose, unlike its two inputs: the built-in declarations are
+ built once per process, while a host policy is resolved per call, so a
+ provider installed after the first purge is honored for any root not yet
+ resolved. ``get_purge_policy`` memoizes per model, so replacing the policy
+ of a root it has already resolved needs a restart -- and that memoization
+ is why rebuilding this small index is off the hot path.
+ """
+ return validate_unique_root_policies(
Review Comment:
my own regression from removing the cache. It's back, keyed on the provider, so one invocation and the same objects on every call.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(retention): let a host install purge policies for its own soft-delete roots [superset]
Posted by "bito-code-review[bot] (via GitHub)" <gi...@apache.org>.
bito-code-review[bot] commented on code in PR #44892:
URL: https://github.com/apache/superset/pull/44892#discussion_r4185308033
##########
tests/unit_tests/tasks/test_deletion_retention.py:
##########
@@ -330,6 +330,50 @@ class UnsupportedModel(SoftDeleteMixin):
engine.dispose()
+@pytest.mark.parametrize("dry_run", [False, True])
+def test_scan_failure_does_not_prevent_supported_models_from_purging(
+ app_config: Config,
+ monkeypatch: pytest.MonkeyPatch,
+ dry_run: bool,
+) -> None:
+ """A root whose scan raises is counted and skipped, not fatal to the run.
+
+ The eligible-id scan sits outside the per-entity handler, so without this
+ one unreadable root would abort the pass and the roots that could have
+ purged never would.
+ """
+ # avoid app-init regression: model helpers require the app_config fixture first.
+ from superset.models.helpers import SoftDeleteMixin
+ from superset.tasks import deletion_retention as task
+
+ supported_models: list[type[SoftDeleteMixin]] = list(task.purge_policy_registry())
Review Comment:
<div>
<div id="suggestion">
<div id="issue"><b>Registry order mismatch</b></div>
<div id="fix">
`supported_models = list(task.purge_policy_registry())` takes the keys of a `Mapping` (purge_policy.py:987), whose order is unrelated to the task's iteration order `_soft_delete_models()` (deletion_retention.py:151). `supported_models[0]` may therefore not be the first model `_purge_roots` processes, so `failing_scan` may never raise and `result["scan_failures"] == 1` fails. The sibling test pins `_registered_subclasses` (line 390) for exactly this reason.
</div>
</div>
<small><i>Code Review Run #1d7448</i></small>
</div>
---
Should Bito avoid suggestions like this for future reviews? (<a href=https://alpha.bito.ai/home/ai-agents/review-rules>Manage Rules</a>)
- [ ] Yes, avoid them
##########
superset/commands/deletion_retention/purge_policy.py:
##########
@@ -860,16 +894,131 @@ def declare(
cleanup_permission=cleanup_dataset_permission,
),
}
- return validate_unique_root_policies(registry.values())
+ return tuple(registry.values())
+
+
+def _host_purge_policies(
+ provider: Callable[[], Any] | None,
+) -> tuple[PurgeEntityPolicy, ...]:
+ """Return the purge policies a host installed for its own roots.
+
+ A host distribution can carry ``SoftDeleteMixin`` entities this package
+ cannot import. The retention task discovers those entities through the
+ mixin registry, so without a policy they reach the cascade as an
+ unsupported model. The host therefore declares their purge behavior and
+ installs it under ``PURGE_POLICIES_FUNC``.
+
+ Host boundary: everything about a host policy is settled here, where it is
+ admitted -- an unavailable provider, a malformed payload, a collision with
+ a root declared in this package, two declarations for one root, and a
+ declaration the shared cleanup could not execute. Each is logged and
+ dropped, so that root is reported as unsupported instead of failing row by
+ row in a scheduled run. A broken host declaration must not take the purge
+ down with it, and must never redefine how a chart, dashboard or dataset is
+ purged.
+ """
+ if provider is None:
+ return ()
+ try:
+ provided: Any = provider()
+ except Exception: # pylint: disable=broad-except
+ logger.exception(
+ "purge_policy: %s is unavailable; keeping built-in roots only",
+ HOST_POLICIES_CONFIG_KEY,
+ )
+ return ()
+ if not isinstance(provided, (list, tuple)) or not all(
+ isinstance(policy, PurgeEntityPolicy) for policy in provided
+ ):
+ logger.error(
+ "purge_policy: %s returned %s; expected a sequence of PurgeEntityPolicy",
+ HOST_POLICIES_CONFIG_KEY,
+ type(provided).__name__,
+ )
+ return ()
+ builtin_roots: frozenset[type[Any]] = frozenset(
+ policy.model for policy in _builtin_purge_policies()
+ )
+ declared: dict[type[Any], int] = {}
+ for policy in provided:
+ declared[policy.model] = declared.get(policy.model, 0) + 1
+ # Which of two declarations for one root is authoritative is undecidable,
+ # and the loser would still delete rows. Dropping both leaves the model
+ # reported as unsupported, which is the recoverable outcome. Resolving it
+ # here also keeps the duplicate away from validate_unique_root_policies,
+ # whose ValueError would abort the whole scheduled run.
+ duplicated: list[type[Any]] = [
+ model for model, count in declared.items() if count > 1
+ ]
+ for model in duplicated:
+ logger.error(
+ "purge_policy: %s declares %d policies for %s; ignoring all of them",
+ HOST_POLICIES_CONFIG_KEY,
+ declared[model],
+ model.__name__,
+ )
+ accepted: list[PurgeEntityPolicy] = []
+ for policy in provided:
+ if policy.model in builtin_roots:
+ logger.error(
+ "purge_policy: host policy for built-in root %s ignored",
+ policy.model.__name__,
+ )
+ continue
+ if policy.model in duplicated:
+ continue
+ try:
+ _validated_policy(policy)
+ except Exception as ex: # pylint: disable=broad-except
+ # Validated at admission rather than when a row is purged: a
+ # declaration the cleanup cannot execute would otherwise fail once
+ # per eligible row, counted as a cascade failure, and stay
+ # invisible until something aged past the window.
+ logger.error(
+ "purge_policy: host policy for %s rejected: %s",
+ policy.model.__name__,
+ ex,
+ )
+ continue
+ accepted.append(policy)
+ return tuple(accepted)
+
+
+def purge_policy_registry() -> Mapping[type[Any], PurgeEntityPolicy]:
+ """Index the built-in purge roots plus any the host installed.
+
+ The index is cached per installed provider, not once per process. Freezing
+ it at first use would pin whatever happened to be installed at that
+ moment -- including nothing at all, for a call made before startup
+ finished -- until a restart. Keying the cache on the provider means it is
+ invoked once however many roots are purged, a host that installs or
+ replaces one is honored, and the ordinary case (no provider) resolves to a
+ single cached index.
+ """
+ provider: Callable[[], Any] | None = (
+ current_app.config.get(HOST_POLICIES_CONFIG_KEY) if has_app_context() else None
+ )
+ return _registry_for(provider)
Review Comment:
<div>
<div id="suggestion">
<div id="issue"><b>Unhashable provider crashes registry</b></div>
<div id="fix">
The old code admitted a malformed provider here: `provider()` ran inside `try/except Exception` and was logged and dropped. Now an unhashable config value (list/dict) reaches `_registry_for`, whose `lru_cache` hashes the key first, so TypeError escapes `purge_policy_registry()` before `_host_purge_policies` can log-and-drop it, contradicting the documented host boundary. Reject non-callables before the cache lookup.
</div>
</div>
<small><i>Code Review Run #1d7448</i></small>
</div>
---
Should Bito avoid suggestions like this for future reviews? (<a href=https://alpha.bito.ai/home/ai-agents/review-rules>Manage Rules</a>)
- [ ] Yes, avoid them
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(retention): let a host install purge policies for its own soft-delete roots [superset]
Posted by "sadpandajoe (via GitHub)" <gi...@apache.org>.
sadpandajoe commented on code in PR #44892:
URL: https://github.com/apache/superset/pull/44892#discussion_r4195557736
##########
superset/commands/deletion_retention/purge_policy.py:
##########
@@ -905,32 +1353,254 @@ def _validated_purge_policy(model: type[Any]) -> PurgeEntityPolicy:
details = (
f"{details}; " if details else ""
) + f"stale_listeners=[{', '.join(coverage.stale_listeners)}]"
- raise RuntimeError(f"Incomplete purge policy for {model.__name__}: {details}")
+ raise RuntimeError(
+ f"Incomplete purge policy for {policy.model.__name__}: {details}"
+ )
return policy
-def _validate_executable_declarations(policy: PurgeEntityPolicy) -> None:
- """Reject executable classifications missing their required action metadata."""
+def _ownership_dependency(
+ policy: PurgeEntityPolicy, related_table: str
+) -> DependencyPolicy | None:
+ """The single owned/association edge attaching *related_table*, if clear."""
+ candidates: tuple[DependencyPolicy, ...] = tuple(
+ dependency
+ for dependency in policy.dependencies
+ if dependency.classification
+ in {DependencyClassification.OWNED, DependencyClassification.ASSOCIATION}
+ and dependency.key.related_table == related_table
+ and dependency.key.direction == "inbound"
+ )
+ return candidates[0] if len(candidates) == 1 else None
+
+
+def _validate_owned_traversal(policy: PurgeEntityPolicy) -> None:
+ """Reject an owned table reachable only through an association.
+
+ The shared cleanup empties associations before owned children, while an
+ owned table's predicate selects its rows *through* its ownership path. If
+ a hop on that path is an association, its rows are already gone when the
+ owned delete runs, so the statement matches nothing.
+
+ Kept where the rest of the ordering rules were dropped because this one
+ can be silent: where foreign keys are enforced the association delete
+ fails loudly, but where they are not -- SQLite -- the root is purged and
+ its descendants are left orphaned with nothing reported. The remedy is to
+ classify the intermediate table as owned, which places it in the same
+ phase as what it leads to.
+ """
+ root_table: str = sa.inspect(policy.model).local_table.name
for dependency in policy.dependencies:
- if (
- dependency.classification is DependencyClassification.LISTENER_EFFECT
- and dependency.listener_action is None
- ):
- raise RuntimeError(
- f"Missing listener action for {dependency.key.describe()}"
- )
- if (
- dependency.classification is DependencyClassification.VERSION_OWNED
- and dependency.version_column is None
+ if dependency.classification is not DependencyClassification.OWNED:
+ continue
+ table_name: str = dependency.key.owner_table
+ visited: set[str] = set()
+ while table_name != root_table and table_name not in visited:
+ visited.add(table_name)
+ hop: DependencyPolicy | None = _ownership_dependency(policy, table_name)
+ if hop is None:
+ # An absent or ambiguous path is reported by coverage, and by
+ # _ownership_edge at execution; not this check's business.
+ break
+ if hop.classification is DependencyClassification.ASSOCIATION:
+ raise RuntimeError(
+ f"Owned dependency {dependency.key.describe()} is reachable "
+ f"only through association {hop.key.describe()}; "
+ "associations are deleted first, so the owned rows would "
+ "be orphaned"
+ )
+ table_name = hop.key.owner_table
+
+
+def _validate_recursive_ownership(policy: PurgeEntityPolicy) -> None:
+ """Reject a self-referencing owned table under the stock cleanup.
+
+ ``delete_owned_children`` issues one statement per declared edge, so a
+ table that owns itself is pruned one level deep. Where foreign keys are
+ enforced the root's own delete then fails and rolls back; where they are
+ not -- SQLite -- the root is purged and its grandchildren are left behind
+ with a dangling parent id and nothing reported. A host declaring a tree
+ supplies cleanup that walks it.
+ """
+ for dependency in policy.dependencies:
+ if dependency.classification is not DependencyClassification.OWNED:
+ continue
+ key: DependencyKey = dependency.key
+ if key.owner_table != key.related_table:
+ continue
+ raise RuntimeError(
+ f"Owned dependency {key.describe()} is self-referencing; the stock "
+ "owned-child cleanup deletes one level, so this policy must supply "
+ "its own delete_owned_children"
+ )
+
+
+def _validate_scanner_requirements(policy: PurgeEntityPolicy) -> None:
+ """Reject a root the scheduled scan cannot page through.
+
+ The retention task selects, windows and orders eligible rows by ``id``,
+ and the cascade pins each row by it. A root keyed on something else -- a
+ UUID primary key with no ``id`` column -- raises inside the scan, outside
+ the per-row error handling, so the run aborts before the remaining roots
+ are reached.
+ """
+ mapper: Mapper[Any] = sa.inspect(policy.model)
+ table: sa.Table = mapper.local_table
+ if "id" not in table.c:
+ raise RuntimeError(
+ f"Purge root {policy.model.__name__} has no 'id' column; the "
+ "scheduled scan pages eligible rows by id"
+ )
+ primary_key: tuple[sa.Column[Any], ...] = tuple(mapper.primary_key)
+ if (
+ len(primary_key) != 1
+ or primary_key[0].name != "id"
+ or not isinstance(primary_key[0].type, sa.Integer)
+ ):
+ # The scan pages by id and the row is then fetched with a scalar
+ # Session.get, so a composite or differently named key matches
+ # nothing: every row fails while a dry run still counts it eligible.
+ raise RuntimeError(
+ f"Purge root {policy.model.__name__} is keyed on "
+ f"({', '.join(column.name for column in primary_key)}); the "
+ "scheduled purge pages an integer 'id' watermark and looks rows "
+ "up by it"
+ )
+ if "deleted_at" not in table.c:
+ raise RuntimeError(
+ f"Purge root {policy.model.__name__} has no 'deleted_at' column; "
+ "every purge path requires the row to be archived first"
+ )
+
+
+#: Classifications the shared cleanup executes as SQL, through
+#: ``_dependency_predicates`` -- which only knows how to read an inbound
+#: foreign key.
+_EXECUTABLE_CLASSIFICATIONS: frozenset[DependencyClassification] = frozenset(
+ {
+ DependencyClassification.OWNED,
+ DependencyClassification.ASSOCIATION,
+ DependencyClassification.BLOCK,
+ }
+)
+
+
+def _validate_version_target(
+ policy: PurgeEntityPolicy, dependency: DependencyPolicy, metadata: sa.MetaData
+) -> None:
+ """Reject a version target whose column does not hold the root's id.
+
+ Version cleanup compares the declared column with the root's id directly,
+ so the column has to be root-relative. Shadow tables are generated and may
+ not exist when a policy is validated, so the check reads their live
+ counterpart -- ``<table>_version`` describes ``<table>``, the naming the
+ shadow declarations already rely on -- and accepts the column when it is
+ the root's own primary key or a foreign key into the root.
+
+ A column naming an intermediate owner, a stage id under a workflow root
+ say, matches history belonging to a different root and leaves this one's
+ behind.
+ """
+ root_table: sa.Table = sa.inspect(policy.model).local_table
+ column: str = cast(str, dependency.version_column)
+ shadow_name: str = dependency.key.related_table
+ suffix: str = "_version"
+ live_name: str = (
+ shadow_name[: -len(suffix)] if shadow_name.endswith(suffix) else shadow_name
+ )
+ live_table: sa.Table | None = metadata.tables.get(live_name)
+ if live_table is not None and column in live_table.c:
+ live_column: sa.Column[Any] = live_table.c[column]
+ primary_key: tuple[sa.Column[Any], ...] = tuple(root_table.primary_key.columns)
+ if live_table is root_table:
+ # On the root's own shadow only its key identifies it. A
+ # self-referencing column such as ``parent_id`` points at *other*
+ # rows of the same root, so cleanup would delete the children's
+ # history and leave the purged row's own behind.
+ if len(primary_key) == 1 and live_column.name == primary_key[0].name:
+ return
+ # Elsewhere the comparison is still against the root's id, so a
+ # foreign key to any other root column matches rows belonging to a
+ # different root.
+ elif len(primary_key) == 1 and any(
+ foreign_key.column.table is root_table
+ and foreign_key.column.name == primary_key[0].name
+ for foreign_key in live_column.foreign_keys
):
- raise RuntimeError(
- f"Missing version target column for {dependency.key.describe()}"
- )
+ return
+ raise RuntimeError(
+ f"Version target {shadow_name}.{column} for {dependency.key.describe()} "
+ f"is not root-relative; cleanup compares it with the "
+ f"{root_table.name} id, so it must be that id or a foreign key to it"
+ )
+
+
+def _validate_dependency_declaration(
+ policy: PurgeEntityPolicy,
+ dependency: DependencyPolicy,
+ metadata: sa.MetaData,
+) -> None:
+ """Reject one declaration whose mistake would not announce itself."""
+ key: DependencyKey = dependency.key
+ if (
+ dependency.classification is DependencyClassification.LISTENER_EFFECT
+ and dependency.listener_action is None
+ ):
+ raise RuntimeError(f"Missing listener action for {key.describe()}")
+ if dependency.classification is DependencyClassification.VERSION_OWNED:
+ if dependency.version_column is None:
+ raise RuntimeError(f"Missing version target column for {key.describe()}")
+ # A version target that is not root-relative deletes another root's
+ # history and leaves this root's behind -- silently, either way.
+ _validate_version_target(policy, dependency, metadata)
Review Comment:
This accepts a root-relative `VERSION_OWNED` target on a child shadow table even when the host root itself is not versioned. `_delete_version_history` in `purge_cascade.py` returns 0 as soon as `version_class(root)` raises `ClassNotVersioned`, so for an unversioned root that owns a versioned child the declared target is never processed: the root and live child rows are purged and the child's history survives, with no failure reported. That is the silent-mistake case this validation is meant to refuse. Should admission reject a `VERSION_OWNED` dependency when the root has no version class, or should version cleanup run the declared targets without it?
##########
superset/commands/deletion_retention/purge_policy.py:
##########
@@ -860,16 +914,408 @@ def declare(
cleanup_permission=cleanup_dataset_permission,
),
}
- return validate_unique_root_policies(registry.values())
+ return tuple(registry.values())
-@lru_cache(maxsize=None)
-def _validated_purge_policy(model: type[Any]) -> PurgeEntityPolicy:
- """Validate and return one root policy without blocking unrelated roots."""
+def _policy_label(policy: Any) -> str:
+ """A log-safe name for a payload whose shape is not yet established."""
+ model: Any = getattr(policy, "model", None)
+ return str(getattr(model, "__name__", type(model).__name__))
+
+
+def _host_policy_payload(provided: Any) -> tuple[Any, ...] | None:
+ """Materialize the provider's payload, or reject it.
+
+ Any iterable is accepted -- a list, a tuple, ``dict.values()``, a
+ generator -- since the documented contract is a sequence of policies, not
+ one particular container. Strings and bytes are excluded because they
+ iterate into characters, which would read as a sequence of bad members
+ rather than the wrong type. Materialized once: a generator cannot be
+ walked twice.
+ """
+ if isinstance(provided, (str, bytes)) or not isinstance(provided, Iterable):
+ logger.error(
+ "purge_policy: %s returned %s; expected an iterable of PurgeEntityPolicy",
+ HOST_POLICIES_CONFIG_KEY,
+ type(provided).__name__,
+ )
+ return None
try:
- policy: PurgeEntityPolicy = purge_policy_registry()[model]
- except KeyError as ex:
- raise ValueError(f"Unsupported purge model: {model.__name__}") from ex
+ return tuple(provided)
+ except Exception: # pylint: disable=broad-except
+ logger.exception(
+ "purge_policy: %s could not be read; keeping built-in roots only",
+ HOST_POLICIES_CONFIG_KEY,
+ )
+ return None
+
+
+def _unqualified_table_conflict(policy: PurgeEntityPolicy) -> str | None:
+ """Return the first table a bare-name lookup would resolve wrongly.
+
+ Discovery records table identities by bare name while a ``MetaData`` keys
+ them by schema, so a declared ``child`` resolves to the default-schema
+ table even where the policy meant ``host.child`` -- and the cleanup would
+ delete rows belonging to unrelated roots. Carrying schema-qualified
+ identities through discovery and execution is the better fix; until then
+ the ambiguous shape is refused rather than silently mis-resolved.
+ """
+ root_table: sa.Table = sa.inspect(policy.model).local_table
+ metadata: sa.MetaData = root_table.metadata
+ keys_by_name: dict[str, list[str]] = {}
+ for table in metadata.tables.values():
+ keys_by_name.setdefault(table.name, []).append(table.key)
+ declared: set[str] = {root_table.name} | {
+ dependency.key.related_table
+ for dependency in policy.dependencies
+ if dependency.classification in _EXECUTABLE_CLASSIFICATIONS
+ }
+ for name in sorted(declared):
+ keys: list[str] = keys_by_name.get(name, [])
+ if len(keys) != 1 or keys[0] != name:
+ return name
+ return None
+
+
+def _admitted_host_policy(
+ candidate: Any, builtin_roots: frozenset[type[Any]], builtin_types: frozenset[str]
+) -> PurgeEntityPolicy | None:
+ """Return *candidate* if it is a usable host policy, else ``None``.
+
+ Every check runs behind one handler rather than guarding each attribute:
+ the payload is host-supplied, so reading ``model`` or hashing it may
+ itself raise -- and an exception escaping here would break
+ ``purge_policy_registry`` for the built-in roots too, which is the
+ opposite of what this boundary is for.
+ """
+ try:
+ if not isinstance(candidate, PurgeEntityPolicy):
+ logger.error(
+ "purge_policy: %s returned a %s; expected PurgeEntityPolicy",
+ HOST_POLICIES_CONFIG_KEY,
+ type(candidate).__name__,
+ )
+ return None
+ if not isinstance(candidate.model, type):
+ # Everything downstream treats the root as a mapped class:
+ # sa.inspect, the registry index, the duplicate count.
+ logger.error(
+ "purge_policy: host policy root %s is not a class",
+ _policy_label(candidate),
+ )
+ return None
+ if candidate.model in builtin_roots:
+ logger.error(
+ "purge_policy: host policy for built-in root %s ignored",
+ candidate.model.__name__,
+ )
+ return None
+ if candidate.entity_type in builtin_types:
+ # Core branches on entity_type -- the dataset impact path, the tag
+ # object type, the dataset permission name -- so a host reusing
+ # one of its names would have that logic pointed at its own rows.
+ logger.error(
+ "purge_policy: host policy for %s claims the reserved entity type %r",
+ candidate.model.__name__,
+ candidate.entity_type,
+ )
+ return None
+ # Validated first, so a table the metadata simply does not contain is
+ # reported as such rather than as an ambiguous name.
+ inherited: list[str] = [
+ name
+ for name, stock in (
+ ("count_dashboard_slices", count_dashboard_slices),
+ ("collect_dangling_chart_uuids", dangling_chart_uuids),
+ )
+ if getattr(candidate, name) is stock
+ ]
+ if inherited:
+ # Cloning a built-in policy with ``replace`` is a natural way to
+ # borrow the generic cleanup, but it also carries these two, which
+ # read ``dashboard_slices`` and the chart table by the root's id.
+ # Left in place on a host root they write a count and a referrer
+ # list for an unrelated entity into the audit record -- wrong
+ # evidence, recorded as fact. The permission pair is inert here by
+ # comparison: both resolve from an entity type a host cannot
+ # claim.
+ raise RuntimeError(
+ f"{', '.join(inherited)} read this package's own tables; a "
+ "host policy supplies its own"
+ )
+ unusable: set[str] = {
+ dependency.classification.value
+ for dependency in candidate.dependencies
+ if dependency.classification
+ in {
+ DependencyClassification.BLOCK,
+ DependencyClassification.LISTENER_EFFECT,
+ }
+ }
+ if unusable:
+ # Both resolve from core's own identities: the stock listener
+ # actions decide what to delete from a built-in entity type, and
+ # blockers are applied by the stock validator. Declared by a host,
+ # they are accepted and then never carried out -- silence rather
+ # than a failure, which is the one outcome worth refusing. A host
+ # blocks by raising PurgeBlockedError from its own validate.
+ raise RuntimeError(
+ f"{', '.join(sorted(unusable))} dependencies cannot be carried "
+ "out for a host root"
+ )
+ _validated_policy(candidate)
+ conflict: str | None = _unqualified_table_conflict(candidate)
+ if conflict is not None:
+ raise RuntimeError(
+ f"table {conflict!r} cannot be identified unambiguously by "
+ "name; schema-qualified roots are not supported"
+ )
+ except Exception as ex: # pylint: disable=broad-except
+ # Validated at admission rather than when a row is purged: a
+ # declaration the cleanup cannot execute would otherwise fail once per
+ # eligible row, counted as a cascade failure, and stay invisible until
+ # something aged past the window.
+ logger.error(
+ "purge_policy: host policy for %s rejected: %s",
+ _policy_label(candidate),
+ ex,
+ )
+ return None
+ return candidate
+
+
+def _host_purge_policies(
+ provider: Callable[[], Any] | None,
+) -> tuple[PurgeEntityPolicy, ...] | None:
+ """Return the purge policies a host installed for its own roots.
+
+ A host distribution can carry ``SoftDeleteMixin`` entities this package
+ cannot import. The retention task discovers those entities through the
+ mixin registry, so without a policy they reach the cascade as an
+ unsupported model. The host therefore declares their purge behavior and
+ installs it under ``PURGE_POLICIES_FUNC``.
+
+ Host boundary: everything about a host policy is settled here, where it is
+ admitted -- an unavailable provider, a payload that is not an iterable of
+ policies, a root that is not a class, a collision with a root or entity
+ type declared in this package, two declarations for one root, and a
+ declaration the shared cleanup could not execute. Each is logged and
+ dropped, so that root is reported as unsupported instead of failing row by
+ row in a scheduled run. A broken host declaration must not take the purge
+ down with it, and must never redefine how a chart, dashboard or dataset is
+ purged.
+
+ Returns ``None`` for the one failure that may be transient -- the provider
+ itself could not be called. The caller then serves the built-in roots
+ *without* publishing, so the next resolution tries again instead of
+ freezing a bad moment in for the life of the process. A payload that is
+ merely malformed is a host bug that will not fix itself, so it is
+ published as an empty result rather than retried on every read.
+ """
+ if provider is None:
+ return ()
+ try:
+ provided: Any = provider()
+ except Exception: # pylint: disable=broad-except
+ logger.exception(
+ "purge_policy: %s is unavailable; keeping built-in roots only",
+ HOST_POLICIES_CONFIG_KEY,
+ )
+ return None
+ candidates: tuple[Any, ...] | None = _host_policy_payload(provided)
Review Comment:
A generator provider that raises while it is being iterated (say it looks up its policies before yielding) fails inside `_host_policy_payload`'s `tuple(provided)`, which returns `None`; that is turned into `()` here and published as a settled registry with no `retry_after`. The host roots then stay unsupported until a restart, whereas a provider that raises when called gets the 60s retry. Should an iteration failure take the same transient path as a call failure? The docstring above ("serves the built-in roots *without* publishing") also no longer matches the 60s deadline the code now publishes.
##########
docs/docs/using-superset/recently-archived.mdx:
##########
@@ -123,7 +123,84 @@ Each skip increments `deletion_retention.unsupported_models.<table>` once per
run. Unsupported models are excluded from `would_purge` and do not count as
`cascade_failures`; supported models continue to use their existing policies.
An empty `unsupported_models` map means no registered models were skipped for
-this reason. This reporting does not add purge support for host-defined models.
+this reason. A root whose eligible-row scan raises — an unreadable column, a
+transient database error — is counted in `scan_failures`, increments
+`deletion_retention.scan_failures.<table>` for that root and gauges
+`deletion_retention.scan_failures` for the run, leaving the remaining roots to
+purge normally. Rows that root had already purged before the failure stay in
+its reported counts, since those deletions are committed.
+
+Reporting a model is not the same as purging it: a host adds purge support for
+its own entities by installing a policy, described next.
+
+A host distribution that carries its own soft-deletable entities can supply
+their purge behavior by installing `PURGE_POLICIES_FUNC`: a callable returning
+one `PurgeEntityPolicy` per root it owns. Installed policies are validated
+against the discovered dependency graph exactly as the built-in ones are, and a
+model that gains an installed policy stops being reported as unsupported. A
+provider that fails, or returns anything other than an iterable of
+`PurgeEntityPolicy` — a list, a tuple, `dict.values()` and a generator all
+qualify — is logged and ignored, leaving the built-in roots unchanged. So is
+a policy whose root is not a class, which everything downstream assumes, and
+one that redeclares a built-in chart, dashboard or dataset root or claims
+one of their `entity_type` names, which the cascade branches on. Two
+declarations for one root are ignored too, so that root is reported as
+unsupported rather than purged by an arbitrary one of them.
+
+Superset guarantees the **frame** around a purge: the locked claim, the
+identity and eligibility predicates, the write-ahead audit record, the
+conditional delete and the blocked/failed accounting. Within that frame a
+policy either uses the shared cleanup callbacks, in which case its declared
+dependencies are the instructions they execute, or supplies its own callbacks
+and owns what happens between the claim and the delete. Either way the frame
+decides whether the row may be purged at all, so no policy can remove a row
+that is not archived and past the window.
+
+Validation is not an attempt to prove that an arbitrary declaration executes
+correctly. A policy is refused when a mistake in it would delete the wrong
+rows or silently skip cleanup, since neither announces itself:
+
+- its root is not keyed on a single `id` column, has no `deleted_at` column,
+ is not a class, or claims an `entity_type` the cascade branches on;
+- a table it names cannot be resolved unambiguously without a schema, so
+ cleanup would resolve it to a different table;
+- a version target is not root-relative, so cleanup would delete another
+ root's history and leave this root's behind;
+- it declares a block or a listener effect, which resolve from Superset's own
+ validator and entity types and so are never carried out for a host root. A
+ host refuses a purge by raising `PurgeBlockedError` from its own validator
+ instead;
+- it keeps the shared owned-child cleanup while declaring a table that owns
+ itself, or one reachable only through an association. The shared cleanup
+ deletes one level and builds each predicate by traversing the ownership
+ path, so without enforced foreign keys the root is purged and the rows
+ below it are orphaned silently. A host declaring either shape supplies its
+ own `delete_owned_children`.
+
+A host root's references from Superset-owned tables are the host's own to
+clean up, in its own callbacks: the tag and permission cleanups Superset runs
+for its roots resolve from entity types a host cannot claim, so they never
+run for one. A host policy also supplies its own `count_dashboard_slices` and
Review Comment:
The stated reason doesn't match the code: for an entity type that isn't `chart`/`dashboard`/`dataset`, `count_dashboard_slices` returns `0` and `dangling_chart_uuids` returns `[]` before reading any table, so the stock callbacks don't record a count or referrer list for an unrelated host entity. An extension author reading this will think they're avoiding audit-record contamination when they are only being forced to write no-op callbacks. Is the intent to require host-supplied callbacks anyway? If so, could the docs (and the matching comment in `_admitted_host_policy`) say that rather than the contamination rationale, or should the `inherited` rejection be dropped?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(retention): let a host install purge policies for its own soft-delete roots [superset]
Posted by "sadpandajoe (via GitHub)" <gi...@apache.org>.
sadpandajoe commented on code in PR #44892:
URL: https://github.com/apache/superset/pull/44892#discussion_r4194661969
##########
superset/commands/deletion_retention/purge_policy.py:
##########
@@ -905,32 +1214,407 @@ def _validated_purge_policy(model: type[Any]) -> PurgeEntityPolicy:
details = (
f"{details}; " if details else ""
) + f"stale_listeners=[{', '.join(coverage.stale_listeners)}]"
- raise RuntimeError(f"Incomplete purge policy for {model.__name__}: {details}")
+ raise RuntimeError(
+ f"Incomplete purge policy for {policy.model.__name__}: {details}"
+ )
return policy
-def _validate_executable_declarations(policy: PurgeEntityPolicy) -> None:
- """Reject executable classifications missing their required action metadata."""
+def _ownership_dependency(
+ policy: PurgeEntityPolicy, related_table: str
+) -> DependencyPolicy | None:
+ """The single owned/association edge attaching *related_table*, if clear."""
+ candidates: tuple[DependencyPolicy, ...] = tuple(
+ dependency
+ for dependency in policy.dependencies
+ if dependency.classification
+ in {DependencyClassification.OWNED, DependencyClassification.ASSOCIATION}
+ and dependency.key.related_table == related_table
+ and dependency.key.direction == "inbound"
+ )
+ return candidates[0] if len(candidates) == 1 else None
+
+
+def _validate_owned_traversal(policy: PurgeEntityPolicy) -> None:
+ """Reject an owned table reachable only through an association.
+
+ ``cascade_hard_delete`` empties associations before owned children, while
+ an owned table's predicate selects its rows *through* its ownership path
+ (see ``_owner_value_select``). If a hop on that path is an association,
+ its rows are already gone when the owned delete runs: the statement
+ matches nothing, leaving the descendants orphaned where foreign keys are
+ unenforced and blocking the root's delete where they are not.
+
+ Refused at declaration time rather than executed. The shape has a
+ remedy -- classify the intermediate table as owned, which places it in
+ the same phase as what it leads to.
+ """
+ root_table: str = sa.inspect(policy.model).local_table.name
for dependency in policy.dependencies:
- if (
- dependency.classification is DependencyClassification.LISTENER_EFFECT
- and dependency.listener_action is None
- ):
+ if dependency.classification is not DependencyClassification.OWNED:
+ continue
+ table_name: str = dependency.key.owner_table
+ visited: set[str] = set()
+ while table_name != root_table and table_name not in visited:
+ visited.add(table_name)
+ hop: DependencyPolicy | None = _ownership_dependency(policy, table_name)
+ if hop is None:
+ # An absent or ambiguous path is reported by coverage and by
+ # _ownership_edge at execution; not this check's business.
+ break
+ if hop.classification is DependencyClassification.ASSOCIATION:
+ raise RuntimeError(
+ f"Owned dependency {dependency.key.describe()} is reachable "
+ f"only through association {hop.key.describe()}; "
+ "associations are deleted first, so the owned rows would "
+ "be orphaned"
+ )
+ table_name = hop.key.owner_table
+
+
+def _tables_share_a_foreign_key(metadata: sa.MetaData, first: str, second: str) -> bool:
+ """Whether either table declares a foreign key into the other."""
+ for owner_name, other_name in ((first, second), (second, first)):
+ table: sa.Table | None = metadata.tables.get(owner_name)
+ if table is None:
+ continue
+ for constraint in table.foreign_key_constraints:
+ if constraint.elements[0].column.table.name == other_name:
+ return True
+ return False
+
+
+def _validate_scanner_requirements(policy: PurgeEntityPolicy) -> None:
+ """Reject a root the scheduled scan cannot page through.
+
+ The retention task selects, windows and orders eligible rows by ``id``,
+ and the cascade pins each row by it. A root keyed on something else -- a
+ UUID primary key with no ``id`` column -- raises inside the scan, outside
+ the per-row error handling, so the run aborts before the remaining roots
+ are reached.
+ """
+ mapper: Mapper[Any] = sa.inspect(policy.model)
+ table: sa.Table = mapper.local_table
+ if "id" not in table.c:
+ raise RuntimeError(
+ f"Purge root {policy.model.__name__} has no 'id' column; the "
+ "scheduled scan pages eligible rows by id"
+ )
+ primary_key: tuple[sa.Column[Any], ...] = tuple(mapper.primary_key)
+ if len(primary_key) != 1 or primary_key[0].name != "id":
+ # The scan pages by id and the row is then fetched with a scalar
+ # Session.get, so a composite or differently named key matches
+ # nothing: every row fails while a dry run still counts it eligible.
+ raise RuntimeError(
+ f"Purge root {policy.model.__name__} is keyed on "
+ f"({', '.join(column.name for column in primary_key)}); the "
+ "scheduled purge looks rows up by a single 'id' primary key"
+ )
+ if "deleted_at" not in table.c:
+ raise RuntimeError(
+ f"Purge root {policy.model.__name__} has no 'deleted_at' column; "
+ "every purge path requires the row to be archived first"
+ )
+
+
+def _validate_recursive_ownership(policy: PurgeEntityPolicy) -> None:
+ """Reject a self-referencing owned table under the stock cleanup.
+
+ ``delete_owned_children`` issues one statement per declared edge, so a
+ table that owns itself is pruned one level deep: a three-level tree
+ leaves its deepest rows behind -- orphaned where foreign keys are
+ unenforced, and blocking the root's own delete where they are not. A host
+ declaring this shape supplies cleanup that walks the whole sub-tree.
+ """
+ for dependency in policy.dependencies:
+ if dependency.classification is not DependencyClassification.OWNED:
+ continue
+ key: DependencyKey = dependency.key
+ if key.owner_table != key.related_table:
+ continue
+ if policy.delete_owned_children is delete_owned_children:
raise RuntimeError(
- f"Missing listener action for {dependency.key.describe()}"
+ f"Owned dependency {key.describe()} is self-referencing; the "
+ "stock owned-child cleanup deletes one level, so this policy "
+ "must supply its own delete_owned_children"
)
- if (
- dependency.classification is DependencyClassification.VERSION_OWNED
- and dependency.version_column is None
+
+
+def _validate_sibling_ordering(
+ policy: PurgeEntityPolicy, classification: DependencyClassification
+) -> None:
+ """Reject same-depth siblings of *classification* that reference one another.
+
+ Owned and association deletes both run through the same depth sort, which
+ is stable -- so two tables at the same depth are deleted in the order they
+ were declared. With a foreign key between them, that order decides the
+ outcome: removing the referenced table first is refused by an enforced
+ constraint and rolls the purge back. Ordering deletes by their own foreign
+ keys would be the richer fix; refusing the shape keeps the engine's
+ contract honest until something needs it.
+
+ Depth is best-effort. A graph the stock traversal cannot measure -- a
+ multi-level tree, say, whose cleanup the host supplies itself -- is left
+ alone rather than rejected for being unmeasurable.
+ """
+ metadata: sa.MetaData = sa.inspect(policy.model).local_table.metadata
+ by_depth: dict[int, list[DependencyPolicy]] = {}
+ for dependency in policy.dependencies:
+ if dependency.classification is not classification:
+ continue
+ try:
+ depth: int = _dependency_owner_depth(policy, dependency.key)
+ except RuntimeError:
+ continue
+ by_depth.setdefault(depth, []).append(dependency)
+ for siblings in by_depth.values():
+ for index, first in enumerate(siblings):
+ for second in siblings[index + 1 :]:
+ if first.key.related_table == second.key.related_table:
+ continue
+ if _tables_share_a_foreign_key(
+ metadata, first.key.related_table, second.key.related_table
+ ):
+ raise RuntimeError(
+ f"{classification.value} dependencies "
+ f"{first.key.describe()} and {second.key.describe()} "
+ "reference each other; these deletes run in "
+ "declaration order, so whether this graph purges "
+ "depends on it"
+ )
+
+
+#: The entity types each stock listener action can act for. Both read
+#: ``policy.entity_type``: tag cleanup maps it to a tag object type, and the
+#: dataset permission name is only built for a dataset. A root outside these
+#: gets a per-row ValueError, or silence, instead of the declared cleanup.
+_LISTENER_ACTION_ENTITY_TYPES: Mapping[ListenerAction, frozenset[str]] = (
+ MappingProxyType(
+ {
+ ListenerAction.DELETE_TAGGED_OBJECTS: frozenset(
+ {"chart", "dashboard", "dataset"}
+ ),
+ ListenerAction.DELETE_DATASET_PERMISSION: frozenset({"dataset"}),
+ }
+ )
+)
+
+
+#: The phase each listener action is dispatched in by the stock callbacks:
+#: tag cleanup runs with the association deletes, permission cleanup once the
+#: entity row is gone and the captured permission name is available.
+_LISTENER_ACTION_PHASES: Mapping[ListenerAction, ExecutionPhase] = MappingProxyType(
+ {
+ ListenerAction.DELETE_TAGGED_OBJECTS: ExecutionPhase.ASSOCIATIONS,
+ ListenerAction.DELETE_DATASET_PERMISSION: ExecutionPhase.POST_DELETE,
+ }
+)
+
+
+#: Classifications the shared cleanup executes as SQL, through
+#: ``_dependency_predicates`` -- which only knows how to read an inbound
+#: foreign key.
+_EXECUTABLE_CLASSIFICATIONS: frozenset[DependencyClassification] = frozenset(
+ {
+ DependencyClassification.OWNED,
+ DependencyClassification.ASSOCIATION,
+ DependencyClassification.BLOCK,
+ }
+)
+
+
+def _validate_listener_dispatch(
+ policy: PurgeEntityPolicy, dependency: DependencyPolicy
+) -> None:
+ """Reject a listener effect no selected callback would carry out.
+
+ Effects are dispatched from inside two stock callbacks -- association
+ cleanup and permission cleanup -- each for one phase, and each resolving
+ its target from the root's entity type. A declaration in another phase, in
+ a phase whose callback this policy replaces, or for an entity type the
+ action cannot act on, is never executed: the root purges and its declared
+ cleanup does not run.
+ """
+ action: ListenerAction = cast(ListenerAction, dependency.listener_action)
+ expected_phase: ExecutionPhase | None = _LISTENER_ACTION_PHASES.get(action)
+ if expected_phase is None:
+ raise RuntimeError(
+ f"Unsupported listener action for {dependency.key.describe()}"
+ )
+ if dependency.phase is not expected_phase:
+ raise RuntimeError(
+ f"Listener effect {dependency.key.describe()} is declared for "
+ f"{cast(ExecutionPhase, dependency.phase).value}, but "
+ f"{action.value} is dispatched only in {expected_phase.value}"
+ )
+ if expected_phase is ExecutionPhase.ASSOCIATIONS:
+ dispatched = policy.delete_associations is delete_associations
+ callback = "delete_associations"
+ else:
+ dispatched = policy.cleanup_permission is cleanup_dataset_permission
+ callback = "cleanup_permission"
+ if not dispatched:
+ raise RuntimeError(
+ f"Listener effect {dependency.key.describe()} is dispatched by the "
+ f"stock {callback} callback, which this policy replaces"
+ )
+ supported: frozenset[str] = _LISTENER_ACTION_ENTITY_TYPES[action]
+ if policy.entity_type not in supported:
+ # The stock actions resolve what to delete from the root's entity
+ # type. An unknown one raises per row for tag cleanup, and yields no
+ # permission name at all for permission cleanup -- which the cascade
+ # then skips silently.
+ raise RuntimeError(
+ f"Listener effect {dependency.key.describe()} runs {action.value}, "
+ f"which supports only the entity types "
+ f"{', '.join(sorted(supported))}; this policy declares "
+ f"{policy.entity_type!r}"
+ )
+
+
+def _validate_version_target(
+ policy: PurgeEntityPolicy, dependency: DependencyPolicy, metadata: sa.MetaData
+) -> None:
+ """Reject a version target whose column does not hold the root's id.
+
+ Version cleanup compares the declared column with the root's id directly,
+ so the column has to be root-relative. Shadow tables are generated and may
+ not exist when a policy is validated, so the check reads their live
+ counterpart -- ``<table>_version`` describes ``<table>``, the naming the
+ shadow declarations already rely on -- and accepts the column when it is
+ the root's own primary key or a foreign key into the root.
+
+ A column naming an intermediate owner, a stage id under a workflow root
+ say, matches history belonging to a different root and leaves this one's
+ behind.
+ """
+ root_table: sa.Table = sa.inspect(policy.model).local_table
+ column: str = cast(str, dependency.version_column)
+ shadow_name: str = dependency.key.related_table
+ suffix: str = "_version"
+ live_name: str = (
+ shadow_name[: -len(suffix)] if shadow_name.endswith(suffix) else shadow_name
+ )
+ live_table: sa.Table | None = metadata.tables.get(live_name)
+ if live_table is not None and column in live_table.c:
+ live_column: sa.Column[Any] = live_table.c[column]
+ primary_key_names: set[str] = {
+ entry.name for entry in root_table.primary_key.columns
+ }
+ if live_table is root_table and live_column.name in primary_key_names:
+ return
+ if any(
Review Comment:
This accepts a foreign key to any column on the root table, but version cleanup compares the target column with the root's `id` (`shadow.c[col] == entity_id`). With roots `(id=10, code=100)` and `(id=20, code=10)` and an owned child keyed on `root.code` declared as the version target, purging root 10 deletes root 20's child history and leaves root 10's behind. Should this require `foreign_key.column` to be the root's single primary-key column?
##########
superset/commands/deletion_retention/purge_policy.py:
##########
@@ -860,16 +896,287 @@ def declare(
cleanup_permission=cleanup_dataset_permission,
),
}
- return validate_unique_root_policies(registry.values())
+ return tuple(registry.values())
-@lru_cache(maxsize=None)
-def _validated_purge_policy(model: type[Any]) -> PurgeEntityPolicy:
- """Validate and return one root policy without blocking unrelated roots."""
+def _policy_label(policy: Any) -> str:
+ """A log-safe name for a payload whose shape is not yet established."""
+ model: Any = getattr(policy, "model", None)
+ return str(getattr(model, "__name__", type(model).__name__))
+
+
+def _host_policy_payload(provided: Any) -> tuple[Any, ...] | None:
+ """Materialize the provider's payload, or reject it.
+
+ Any iterable is accepted -- a list, a tuple, ``dict.values()``, a
+ generator -- since the documented contract is a sequence of policies, not
+ one particular container. Strings and bytes are excluded because they
+ iterate into characters, which would read as a sequence of bad members
+ rather than the wrong type. Materialized once: a generator cannot be
+ walked twice.
+ """
+ if isinstance(provided, (str, bytes)) or not isinstance(provided, Iterable):
+ logger.error(
+ "purge_policy: %s returned %s; expected an iterable of PurgeEntityPolicy",
+ HOST_POLICIES_CONFIG_KEY,
+ type(provided).__name__,
+ )
+ return None
try:
- policy: PurgeEntityPolicy = purge_policy_registry()[model]
- except KeyError as ex:
- raise ValueError(f"Unsupported purge model: {model.__name__}") from ex
+ return tuple(provided)
+ except Exception: # pylint: disable=broad-except
+ logger.exception(
+ "purge_policy: %s could not be read; keeping built-in roots only",
+ HOST_POLICIES_CONFIG_KEY,
+ )
+ return None
+
+
+def _unqualified_table_conflict(policy: PurgeEntityPolicy) -> str | None:
+ """Return the first table a bare-name lookup would resolve wrongly.
+
+ Discovery records table identities by bare name while a ``MetaData`` keys
+ them by schema, so a declared ``child`` resolves to the default-schema
+ table even where the policy meant ``host.child`` -- and the cleanup would
+ delete rows belonging to unrelated roots. Carrying schema-qualified
+ identities through discovery and execution is the better fix; until then
+ the ambiguous shape is refused rather than silently mis-resolved.
+ """
+ root_table: sa.Table = sa.inspect(policy.model).local_table
+ metadata: sa.MetaData = root_table.metadata
+ keys_by_name: dict[str, list[str]] = {}
+ for table in metadata.tables.values():
+ keys_by_name.setdefault(table.name, []).append(table.key)
+ declared: set[str] = {root_table.name} | {
+ dependency.key.related_table
+ for dependency in policy.dependencies
+ if dependency.classification in _EXECUTABLE_CLASSIFICATIONS
+ }
+ for name in sorted(declared):
+ keys: list[str] = keys_by_name.get(name, [])
+ if len(keys) != 1 or keys[0] != name:
+ return name
+ return None
+
+
+def _admitted_host_policy(
+ candidate: Any, builtin_roots: frozenset[type[Any]], builtin_types: frozenset[str]
+) -> PurgeEntityPolicy | None:
+ """Return *candidate* if it is a usable host policy, else ``None``.
+
+ Every check runs behind one handler rather than guarding each attribute:
+ the payload is host-supplied, so reading ``model`` or hashing it may
+ itself raise -- and an exception escaping here would break
+ ``purge_policy_registry`` for the built-in roots too, which is the
+ opposite of what this boundary is for.
+ """
+ try:
+ if not isinstance(candidate, PurgeEntityPolicy):
+ logger.error(
+ "purge_policy: %s returned a %s; expected PurgeEntityPolicy",
+ HOST_POLICIES_CONFIG_KEY,
+ type(candidate).__name__,
+ )
+ return None
+ if not isinstance(candidate.model, type):
+ # Everything downstream treats the root as a mapped class:
+ # sa.inspect, the registry index, the duplicate count.
+ logger.error(
+ "purge_policy: host policy root %s is not a class",
+ _policy_label(candidate),
+ )
+ return None
+ if candidate.model in builtin_roots:
+ logger.error(
+ "purge_policy: host policy for built-in root %s ignored",
+ candidate.model.__name__,
+ )
+ return None
+ if candidate.entity_type in builtin_types:
+ # Core branches on entity_type -- the dataset impact path, the tag
+ # object type, the dataset permission name -- so a host reusing
+ # one of its names would have that logic pointed at its own rows.
+ logger.error(
+ "purge_policy: host policy for %s claims the reserved entity type %r",
+ candidate.model.__name__,
+ candidate.entity_type,
+ )
+ return None
+ # Validated first, so a table the metadata simply does not contain is
+ # reported as such rather than as an ambiguous name.
+ _validated_policy(candidate)
+ conflict: str | None = _unqualified_table_conflict(candidate)
+ if conflict is not None:
+ raise RuntimeError(
+ f"table {conflict!r} cannot be identified unambiguously by "
+ "name; schema-qualified roots are not supported"
+ )
+ except Exception as ex: # pylint: disable=broad-except
+ # Validated at admission rather than when a row is purged: a
+ # declaration the cleanup cannot execute would otherwise fail once per
+ # eligible row, counted as a cascade failure, and stay invisible until
+ # something aged past the window.
+ logger.error(
+ "purge_policy: host policy for %s rejected: %s",
+ _policy_label(candidate),
+ ex,
+ )
+ return None
+ return candidate
+
+
+def _host_purge_policies(
+ provider: Callable[[], Any] | None,
+) -> tuple[PurgeEntityPolicy, ...]:
+ """Return the purge policies a host installed for its own roots.
+
+ A host distribution can carry ``SoftDeleteMixin`` entities this package
+ cannot import. The retention task discovers those entities through the
+ mixin registry, so without a policy they reach the cascade as an
+ unsupported model. The host therefore declares their purge behavior and
+ installs it under ``PURGE_POLICIES_FUNC``.
+
+ Host boundary: everything about a host policy is settled here, where it is
+ admitted -- an unavailable provider, a payload that is not an iterable of
+ policies, a root that is not a class, a collision with a root or entity
+ type declared in this package, two declarations for one root, and a
+ declaration the shared cleanup could not execute. Each is logged and
+ dropped, so that root is reported as unsupported instead of failing row by
+ row in a scheduled run. A broken host declaration must not take the purge
+ down with it, and must never redefine how a chart, dashboard or dataset is
+ purged.
+ """
+ if provider is None:
+ return ()
+ try:
+ provided: Any = provider()
+ except Exception: # pylint: disable=broad-except
+ logger.exception(
+ "purge_policy: %s is unavailable; keeping built-in roots only",
+ HOST_POLICIES_CONFIG_KEY,
+ )
+ return ()
+ candidates: tuple[Any, ...] | None = _host_policy_payload(provided)
+ if candidates is None:
+ return ()
+
+ builtin_policies: tuple[PurgeEntityPolicy, ...] = _builtin_purge_policies()
+ builtin_roots: frozenset[type[Any]] = frozenset(
+ policy.model for policy in builtin_policies
+ )
+ builtin_types: frozenset[str] = frozenset(
+ policy.entity_type for policy in builtin_policies
+ )
+ admitted: list[PurgeEntityPolicy] = [
+ policy
+ for candidate in candidates
+ if (policy := _admitted_host_policy(candidate, builtin_roots, builtin_types))
+ is not None
+ ]
+
+ # Counted after admission, where every root is known to be a class and so
+ # hashable. Which of two declarations for one root is authoritative is
+ # undecidable, and the loser would still delete rows, so both are dropped:
+ # the root is reported as unsupported, which is the recoverable outcome.
+ # It also keeps the duplicate away from validate_unique_root_policies,
+ # whose ValueError would abort the whole scheduled run.
+ counts: dict[type[Any], int] = {}
+ for policy in admitted:
+ counts[policy.model] = counts.get(policy.model, 0) + 1
+ for model, count in counts.items():
+ if count > 1:
+ logger.error(
+ "purge_policy: %s declares %d policies for %s; ignoring all of them",
+ HOST_POLICIES_CONFIG_KEY,
+ count,
+ model.__name__,
+ )
+ return tuple(policy for policy in admitted if counts[policy.model] == 1)
+
+
+#: Sentinel distinguishing "no index resolved yet" from an index resolved for
+#: no provider, which is the ordinary case.
+_UNRESOLVED: Any = object()
+
+
+@dataclass
+class _ResolvedRegistry:
+ """The index built for one provider, with the roots validated so far.
+
+ Compared by provider *identity* rather than memoized with ``lru_cache``: a
+ cache keyed on the provider would hash it, and a host callable
+ implemented as a mutable dataclass instance is unhashable. That would
+ raise here -- before the host boundary could isolate the host's mistake --
+ and stop the built-in roots purging too.
+ """
+
+ provider: Any = _UNRESOLVED
+ registry: Mapping[type[Any], PurgeEntityPolicy] = field(
+ default_factory=lambda: MappingProxyType({})
+ )
+ validated: set[type[Any]] = field(default_factory=set)
+
+
+_RESOLVED: _ResolvedRegistry = _ResolvedRegistry()
+
+#: Serializes rebuilds so concurrent first uses invoke the provider once.
+_RESOLVE_LOCK: threading.Lock = threading.Lock()
+
+
+def _resolved_registry() -> _ResolvedRegistry:
+ """Return the state resolved for the installed provider, building it once.
+
+ Readers take a snapshot and the rebuild publishes a new one, so a caller
+ always sees an index and its validation record together. Rebuilds are
+ serialized and re-checked under the lock: two concurrent first uses would
+ otherwise both invoke the provider, and the slower one would publish over
+ the other -- discarding host roots it had admitted if that second
+ invocation happened to fail transiently.
+ """
+ provider: Callable[[], Any] | None = (
+ current_app.config.get(HOST_POLICIES_CONFIG_KEY) if has_app_context() else None
+ )
+ resolved: _ResolvedRegistry = _RESOLVED
+ if resolved.provider is provider:
+ return resolved
+ with _RESOLVE_LOCK:
Review Comment:
The provider runs here while the non-reentrant `_RESOLVE_LOCK` is held and before the new state is published. If a host provider builds its policy from a built-in one (for example `replace(get_purge_policy(Slice), model=HostRoot, ...)`), `get_purge_policy` re-enters `_resolved_registry()` and blocks forever on the same lock, stalling the scheduled purge and every later registry lookup. Should provider construction run outside the lock, or should re-entry be detected and fail closed?
##########
superset/commands/deletion_retention/purge_policy.py:
##########
@@ -905,32 +1214,407 @@ def _validated_purge_policy(model: type[Any]) -> PurgeEntityPolicy:
details = (
f"{details}; " if details else ""
) + f"stale_listeners=[{', '.join(coverage.stale_listeners)}]"
- raise RuntimeError(f"Incomplete purge policy for {model.__name__}: {details}")
+ raise RuntimeError(
+ f"Incomplete purge policy for {policy.model.__name__}: {details}"
+ )
return policy
-def _validate_executable_declarations(policy: PurgeEntityPolicy) -> None:
- """Reject executable classifications missing their required action metadata."""
+def _ownership_dependency(
+ policy: PurgeEntityPolicy, related_table: str
+) -> DependencyPolicy | None:
+ """The single owned/association edge attaching *related_table*, if clear."""
+ candidates: tuple[DependencyPolicy, ...] = tuple(
+ dependency
+ for dependency in policy.dependencies
+ if dependency.classification
+ in {DependencyClassification.OWNED, DependencyClassification.ASSOCIATION}
+ and dependency.key.related_table == related_table
+ and dependency.key.direction == "inbound"
+ )
+ return candidates[0] if len(candidates) == 1 else None
+
+
+def _validate_owned_traversal(policy: PurgeEntityPolicy) -> None:
+ """Reject an owned table reachable only through an association.
+
+ ``cascade_hard_delete`` empties associations before owned children, while
+ an owned table's predicate selects its rows *through* its ownership path
+ (see ``_owner_value_select``). If a hop on that path is an association,
+ its rows are already gone when the owned delete runs: the statement
+ matches nothing, leaving the descendants orphaned where foreign keys are
+ unenforced and blocking the root's delete where they are not.
+
+ Refused at declaration time rather than executed. The shape has a
+ remedy -- classify the intermediate table as owned, which places it in
+ the same phase as what it leads to.
+ """
+ root_table: str = sa.inspect(policy.model).local_table.name
for dependency in policy.dependencies:
- if (
- dependency.classification is DependencyClassification.LISTENER_EFFECT
- and dependency.listener_action is None
- ):
+ if dependency.classification is not DependencyClassification.OWNED:
+ continue
+ table_name: str = dependency.key.owner_table
+ visited: set[str] = set()
+ while table_name != root_table and table_name not in visited:
+ visited.add(table_name)
+ hop: DependencyPolicy | None = _ownership_dependency(policy, table_name)
+ if hop is None:
+ # An absent or ambiguous path is reported by coverage and by
+ # _ownership_edge at execution; not this check's business.
+ break
+ if hop.classification is DependencyClassification.ASSOCIATION:
+ raise RuntimeError(
+ f"Owned dependency {dependency.key.describe()} is reachable "
+ f"only through association {hop.key.describe()}; "
+ "associations are deleted first, so the owned rows would "
+ "be orphaned"
+ )
+ table_name = hop.key.owner_table
+
+
+def _tables_share_a_foreign_key(metadata: sa.MetaData, first: str, second: str) -> bool:
+ """Whether either table declares a foreign key into the other."""
+ for owner_name, other_name in ((first, second), (second, first)):
+ table: sa.Table | None = metadata.tables.get(owner_name)
+ if table is None:
+ continue
+ for constraint in table.foreign_key_constraints:
+ if constraint.elements[0].column.table.name == other_name:
+ return True
+ return False
+
+
+def _validate_scanner_requirements(policy: PurgeEntityPolicy) -> None:
+ """Reject a root the scheduled scan cannot page through.
+
+ The retention task selects, windows and orders eligible rows by ``id``,
+ and the cascade pins each row by it. A root keyed on something else -- a
+ UUID primary key with no ``id`` column -- raises inside the scan, outside
+ the per-row error handling, so the run aborts before the remaining roots
+ are reached.
+ """
+ mapper: Mapper[Any] = sa.inspect(policy.model)
+ table: sa.Table = mapper.local_table
+ if "id" not in table.c:
+ raise RuntimeError(
+ f"Purge root {policy.model.__name__} has no 'id' column; the "
+ "scheduled scan pages eligible rows by id"
+ )
+ primary_key: tuple[sa.Column[Any], ...] = tuple(mapper.primary_key)
+ if len(primary_key) != 1 or primary_key[0].name != "id":
Review Comment:
A host root with a single UUID primary key named `id` passes this check, but the scan in `_iter_eligible_ids` starts its watermark at integer `0` and filters `id > 0`, which fails on PostgreSQL. Its eligible rows are never purged and every run reports a scan failure. Could admission also require an integer `id` primary key?
##########
superset/commands/deletion_retention/purge_policy.py:
##########
@@ -905,32 +1214,407 @@ def _validated_purge_policy(model: type[Any]) -> PurgeEntityPolicy:
details = (
f"{details}; " if details else ""
) + f"stale_listeners=[{', '.join(coverage.stale_listeners)}]"
- raise RuntimeError(f"Incomplete purge policy for {model.__name__}: {details}")
+ raise RuntimeError(
+ f"Incomplete purge policy for {policy.model.__name__}: {details}"
+ )
return policy
-def _validate_executable_declarations(policy: PurgeEntityPolicy) -> None:
- """Reject executable classifications missing their required action metadata."""
+def _ownership_dependency(
+ policy: PurgeEntityPolicy, related_table: str
+) -> DependencyPolicy | None:
+ """The single owned/association edge attaching *related_table*, if clear."""
+ candidates: tuple[DependencyPolicy, ...] = tuple(
+ dependency
+ for dependency in policy.dependencies
+ if dependency.classification
+ in {DependencyClassification.OWNED, DependencyClassification.ASSOCIATION}
+ and dependency.key.related_table == related_table
+ and dependency.key.direction == "inbound"
+ )
+ return candidates[0] if len(candidates) == 1 else None
+
+
+def _validate_owned_traversal(policy: PurgeEntityPolicy) -> None:
+ """Reject an owned table reachable only through an association.
+
+ ``cascade_hard_delete`` empties associations before owned children, while
+ an owned table's predicate selects its rows *through* its ownership path
+ (see ``_owner_value_select``). If a hop on that path is an association,
+ its rows are already gone when the owned delete runs: the statement
+ matches nothing, leaving the descendants orphaned where foreign keys are
+ unenforced and blocking the root's delete where they are not.
+
+ Refused at declaration time rather than executed. The shape has a
+ remedy -- classify the intermediate table as owned, which places it in
+ the same phase as what it leads to.
+ """
+ root_table: str = sa.inspect(policy.model).local_table.name
for dependency in policy.dependencies:
- if (
- dependency.classification is DependencyClassification.LISTENER_EFFECT
- and dependency.listener_action is None
- ):
+ if dependency.classification is not DependencyClassification.OWNED:
+ continue
+ table_name: str = dependency.key.owner_table
+ visited: set[str] = set()
+ while table_name != root_table and table_name not in visited:
+ visited.add(table_name)
+ hop: DependencyPolicy | None = _ownership_dependency(policy, table_name)
+ if hop is None:
Review Comment:
An owned table with two ownership parents that also owns children (root `R`, owned `A` and `B`, owned `X` reached from both `A` and `B`, owned `Y` under `X`) is admitted: this loop breaks on the ambiguous hop and the sibling-ordering check skips what it cannot measure. At execution `delete_owned_children` raises `Expected one ownership path to x`, so every eligible row fails with a cascade failure on every run. Should admission reject an owned table with more than one ownership path instead of deferring to execution?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(retention): let a host install purge policies for its own soft-delete roots [superset]
Posted by "alexandrusoare (via GitHub)" <gi...@apache.org>.
alexandrusoare commented on code in PR #44892:
URL: https://github.com/apache/superset/pull/44892#discussion_r4165609720
##########
docs/docs/using-superset/recently-archived.mdx:
##########
@@ -123,7 +123,18 @@ Each skip increments `deletion_retention.unsupported_models.<table>` once per
run. Unsupported models are excluded from `would_purge` and do not count as
`cascade_failures`; supported models continue to use their existing policies.
An empty `unsupported_models` map means no registered models were skipped for
-this reason. This reporting does not add purge support for host-defined models.
+this reason. This reporting does not itself add purge support for host-defined
+models.
+
+A host distribution that carries its own soft-deletable entities can supply
+their purge behavior by installing `PURGE_POLICIES_FUNC`: a callable returning
+one `PurgeEntityPolicy` per root it owns. Installed policies are validated
+against the discovered dependency graph exactly as the built-in ones are, and a
+model that gains an installed policy stops being reported as unsupported. A
+provider that fails, returns something other than a sequence of policies, or
+redeclares a built-in chart, dashboard or dataset root is logged and ignored,
+leaving the built-in roots unchanged. Hosts can require
+`superset.versioning.utils.HOST_POLICY_API_VERSION` before installing.
Review Comment:
Good catch, capability check sounds great
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(retention): let a host install purge policies for its own soft-delete roots [superset]
Posted by "alexandrusoare (via GitHub)" <gi...@apache.org>.
alexandrusoare commented on code in PR #44892:
URL: https://github.com/apache/superset/pull/44892#discussion_r4183750454
##########
superset/commands/deletion_retention/purge_policy.py:
##########
@@ -329,34 +363,34 @@ def _discover_table_dependencies(
if constraint.elements[0].column.table is table:
discovered.add(_fk_key(table, constraint, "inbound"))
discovered.update(
- _discover_recursive_table_dependencies(
- table,
- recursive_tables=recursive_tables,
- visited=next_visited,
- )
+ _discover_recursive_table_dependencies(table, recursive_tables, seen)
)
return frozenset(discovered)
def _discover_recursive_table_dependencies(
table: sa.Table,
- *,
recursive_tables: frozenset[str],
- visited: frozenset[str],
+ seen: _Traversal,
) -> frozenset[DependencyKey]:
- """Discover dependencies for named recursive tables not already visited."""
+ """Discover dependencies for named recursive tables not already visited.
+
+ Sorted so a traversal is reproducible, and driven off the shared
+ bookkeeping so each table is walked at most once per kind of walk.
+ """
discovered: set[DependencyKey] = set()
- for recursive_table_name in recursive_tables - visited:
+ for recursive_table_name in sorted(recursive_tables - seen.tables):
+ # Re-checked inside the loop: the set grows as the walk descends, so
+ # a table a nested call already covered is skipped rather than
+ # re-entered to return nothing.
+ if recursive_table_name in seen.tables:
Review Comment:
Added tests
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(retention): let a host install purge policies for its own soft-delete roots [superset]
Posted by "alexandrusoare (via GitHub)" <gi...@apache.org>.
alexandrusoare commented on code in PR #44892:
URL: https://github.com/apache/superset/pull/44892#discussion_r4206022624
##########
superset/commands/deletion_retention/purge_policy.py:
##########
@@ -905,32 +1398,283 @@ def _validated_purge_policy(model: type[Any]) -> PurgeEntityPolicy:
details = (
f"{details}; " if details else ""
) + f"stale_listeners=[{', '.join(coverage.stale_listeners)}]"
- raise RuntimeError(f"Incomplete purge policy for {model.__name__}: {details}")
+ raise RuntimeError(
+ f"Incomplete purge policy for {policy.model.__name__}: {details}"
+ )
return policy
-def _validate_executable_declarations(policy: PurgeEntityPolicy) -> None:
- """Reject executable classifications missing their required action metadata."""
+def _ownership_dependency(
+ policy: PurgeEntityPolicy, related_table: str
+) -> DependencyPolicy | None:
+ """The single owned/association edge attaching *related_table*, if clear."""
+ candidates: tuple[DependencyPolicy, ...] = tuple(
+ dependency
+ for dependency in policy.dependencies
+ if dependency.classification
+ in {DependencyClassification.OWNED, DependencyClassification.ASSOCIATION}
+ and dependency.key.related_table == related_table
+ and dependency.key.direction == "inbound"
+ )
+ return candidates[0] if len(candidates) == 1 else None
+
+
+def _validate_owned_traversal(policy: PurgeEntityPolicy) -> None:
+ """Reject an owned table reachable only through an association.
+
+ The shared cleanup empties associations before owned children, while an
+ owned table's predicate selects its rows *through* its ownership path. If
+ a hop on that path is an association, its rows are already gone when the
+ owned delete runs, so the statement matches nothing.
+
+ Kept where the rest of the ordering rules were dropped because this one
+ can be silent: where foreign keys are enforced the association delete
+ fails loudly, but where they are not -- SQLite -- the root is purged and
+ its descendants are left orphaned with nothing reported. The remedy is to
+ classify the intermediate table as owned, which places it in the same
+ phase as what it leads to.
+ """
+ root_table: str = sa.inspect(policy.model).local_table.name
for dependency in policy.dependencies:
- if (
- dependency.classification is DependencyClassification.LISTENER_EFFECT
- and dependency.listener_action is None
- ):
- raise RuntimeError(
- f"Missing listener action for {dependency.key.describe()}"
- )
- if (
- dependency.classification is DependencyClassification.VERSION_OWNED
- and dependency.version_column is None
+ if dependency.classification is not DependencyClassification.OWNED:
+ continue
+ table_name: str = dependency.key.owner_table
+ visited: set[str] = set()
+ while table_name != root_table and table_name not in visited:
+ visited.add(table_name)
+ hop: DependencyPolicy | None = _ownership_dependency(policy, table_name)
+ if hop is None:
+ # An absent or ambiguous path is reported by coverage, and by
+ # _ownership_edge at execution; not this check's business.
+ break
+ if hop.classification is DependencyClassification.ASSOCIATION:
+ raise RuntimeError(
+ f"Owned dependency {dependency.key.describe()} is reachable "
+ f"only through association {hop.key.describe()}; "
+ "associations are deleted first, so the owned rows would "
+ "be orphaned"
+ )
+ table_name = hop.key.owner_table
+
+
+def _validate_recursive_ownership(policy: PurgeEntityPolicy) -> None:
+ """Reject a self-referencing owned table under one-level cleanup.
+
+ Cleanup that issues one statement per declared edge prunes a table that
+ owns itself one level deep. Where foreign keys are enforced the root's own
+ delete then fails and rolls back; where they are not -- SQLite -- the root
+ is purged and its grandchildren are left behind with a dangling parent id
+ and nothing reported. A policy declaring a tree walks it itself and says
+ so with ``walks_own_subtree``.
+ """
+ for dependency in policy.dependencies:
+ if dependency.classification is not DependencyClassification.OWNED:
+ continue
+ key: DependencyKey = dependency.key
+ if key.owner_table != key.related_table:
+ continue
+ raise RuntimeError(
+ f"Owned dependency {key.describe()} is self-referencing; the stock "
+ "owned-child cleanup deletes one level, so this policy must supply "
+ "its own delete_owned_children"
+ )
+
+
+def _validate_scanner_requirements(policy: PurgeEntityPolicy) -> None:
+ """Reject a root the scheduled scan cannot page through.
+
+ The retention task selects, windows and orders eligible rows by ``id``,
+ and the cascade pins each row by it. A root keyed on something else -- a
+ UUID primary key with no ``id`` column -- raises inside the scan, outside
+ the per-row error handling, so the run aborts before the remaining roots
+ are reached.
+ """
+ mapper: Mapper[Any] = sa.inspect(policy.model)
+ table: sa.Table = mapper.local_table
+ if "id" not in table.c:
+ raise RuntimeError(
+ f"Purge root {policy.model.__name__} has no 'id' column; the "
+ "scheduled scan pages eligible rows by id"
+ )
+ primary_key: tuple[sa.Column[Any], ...] = tuple(mapper.primary_key)
Review Comment:
it fails per row and is counted as a cascade failure with a traceback, which is the loud category the PR says it doesn't guard. We're not trying to cover every host shape — added to the known-unsupported list instead
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(retention): let a host install purge policies for its own soft-delete roots [superset]
Posted by "sadpandajoe (via GitHub)" <gi...@apache.org>.
sadpandajoe commented on code in PR #44892:
URL: https://github.com/apache/superset/pull/44892#discussion_r4208416168
##########
superset/commands/deletion_retention/purge_policy.py:
##########
@@ -905,32 +1549,380 @@ def _validated_purge_policy(model: type[Any]) -> PurgeEntityPolicy:
details = (
f"{details}; " if details else ""
) + f"stale_listeners=[{', '.join(coverage.stale_listeners)}]"
- raise RuntimeError(f"Incomplete purge policy for {model.__name__}: {details}")
+ raise RuntimeError(
+ f"Incomplete purge policy for {policy.model.__name__}: {details}"
+ )
return policy
-def _validate_executable_declarations(policy: PurgeEntityPolicy) -> None:
- """Reject executable classifications missing their required action metadata."""
+def _ownership_dependency(
+ policy: PurgeEntityPolicy, related_table: str
+) -> DependencyPolicy | None:
+ """The single owned/association edge attaching *related_table*, if clear."""
+ candidates: tuple[DependencyPolicy, ...] = tuple(
+ dependency
+ for dependency in policy.dependencies
+ if dependency.classification
+ in {DependencyClassification.OWNED, DependencyClassification.ASSOCIATION}
+ and dependency.key.related_table == related_table
+ and dependency.key.direction == "inbound"
+ )
+ return candidates[0] if len(candidates) == 1 else None
+
+
+def _validate_owned_traversal(policy: PurgeEntityPolicy) -> None:
+ """Reject an owned table reachable only through an association.
+
+ The shared cleanup empties associations before owned children, while an
+ owned table's predicate selects its rows *through* its ownership path. If
+ a hop on that path is an association, its rows are already gone when the
+ owned delete runs, so the statement matches nothing.
+
+ Kept where the rest of the ordering rules were dropped because this one
+ can be silent: where foreign keys are enforced the association delete
+ fails loudly, but where they are not -- SQLite -- the root is purged and
+ its descendants are left orphaned with nothing reported. The remedy is to
+ classify the intermediate table as owned, which places it in the same
+ phase as what it leads to.
+ """
+ root_table: str = sa.inspect(policy.model).local_table.name
+ for dependency in policy.dependencies:
+ if dependency.classification is not DependencyClassification.OWNED:
+ continue
+ table_name: str = dependency.key.owner_table
+ visited: set[str] = set()
+ while table_name != root_table and table_name not in visited:
+ visited.add(table_name)
+ hop: DependencyPolicy | None = _ownership_dependency(policy, table_name)
+ if hop is None:
+ # An absent or ambiguous path is reported by coverage, and by
+ # _ownership_edge at execution; not this check's business.
+ break
+ if hop.classification is DependencyClassification.ASSOCIATION:
+ raise RuntimeError(
+ f"Owned dependency {dependency.key.describe()} is reachable "
+ f"only through association {hop.key.describe()}; "
+ "associations are deleted first, so the owned rows would "
+ "be orphaned"
+ )
+ table_name = hop.key.owner_table
+
+
+def _validate_recursive_ownership(policy: PurgeEntityPolicy) -> None:
+ """Reject a self-referencing owned table under one-level cleanup.
+
+ Cleanup that issues one statement per declared edge prunes a table that
+ owns itself one level deep. Where foreign keys are enforced the root's own
+ delete then fails and rolls back; where they are not -- SQLite -- the root
+ is purged and its grandchildren are left behind with a dangling parent id
+ and nothing reported. A policy declaring a tree walks it itself and says
+ so with ``walks_own_subtree``.
+ """
+ for dependency in policy.dependencies:
+ if dependency.classification is not DependencyClassification.OWNED:
+ continue
+ key: DependencyKey = dependency.key
+ if key.owner_table != key.related_table:
Review Comment:
A host policy with a root↔child foreign-key cycle (`root.link_id → link.id` and `link.root_id → root.id`), with both edges classified `OWNED`, passes admission: this guard only refuses an owned edge whose owner and related table are the same, and `_validate_owned_traversal` finds an ownership path for each. The stock owned cleanup then runs the back-edge as `DELETE FROM root WHERE link_id IN (SELECT id FROM link WHERE root_id = :id)`, so purging archived root 1 also deletes live root 2 if it references link 10 — with no eligibility predicate and no audit record, and it commits cleanly even with enforced FKs. The `ASSOCIATION` spelling of an edge into the root's table is refused unconditionally; should an `OWNED` edge whose related table is the root's own table be refused the same way, as the docs list it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(retention): let a host install purge policies for its own soft-delete roots [superset]
Posted by "alexandrusoare (via GitHub)" <gi...@apache.org>.
alexandrusoare commented on PR #44892:
URL: https://github.com/apache/superset/pull/44892#issuecomment-5993729970
> * Is it acceptable for host policies to be resolved per call, and then memoized forever in get_purge_policy, rather than frozen once at app startup?
You're right that it was the wrong half of each. The index is now cached per provider, so the provider runs once per process, and validation is remembered per declaration rather than per model, so a replaced policy gets revalidated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(retention): let a host install purge policies for its own soft-delete roots [superset]
Posted by "bito-code-review[bot] (via GitHub)" <gi...@apache.org>.
bito-code-review[bot] commented on PR #44892:
URL: https://github.com/apache/superset/pull/44892#issuecomment-5999991178
AI Code Review is in progress (usually takes 3 to 15 minutes unless it's a very large PR).
<details>
<summary>
Bito Usage Guide
</summary>
**Commands**
Type the following command in the pull request comment and save the comment.
- `/review` - Manually triggers an incremental AI Review.
- `/review full` - Manually triggers a full AI Review.
- `/pause` - Pauses automatic reviews on this pull request.
- `/resume` - Resumes automatic reviews.
- `/resolve` - Marks all Bito-posted review comments as resolved.
- `/abort` - Cancels all in-progress reviews.
Refer to the <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/available-commands" target="_blank" rel="noopener noreferrer">documentation</a> for additional commands.
**Configuration**
This repository uses `Superset` You can customize the agent settings <a href="https://alpha.bito.ai/home/ai-agents/code-review-agent" target="_blank" rel="noopener noreferrer">here</a> or contact your Bito workspace admin at evan@preset.io.
**Documentation & Help**
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/getting-started/install-run-using-bito-cloud/create-or-customize-an-agent-instance" target="_blank" rel="noopener noreferrer">Customize agent settings</a>
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/implementing-custom-code-review-rules" target="_blank" rel="noopener noreferrer">Review rules</a>
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent" target="_blank" rel="noopener noreferrer">General documentation</a>
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/faqs" target="_blank" rel="noopener noreferrer">FAQ</a>
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(retention): let a host install purge policies for its own soft-delete roots [superset]
Posted by "sadpandajoe (via GitHub)" <gi...@apache.org>.
sadpandajoe commented on code in PR #44892:
URL: https://github.com/apache/superset/pull/44892#discussion_r4199522861
##########
superset/commands/deletion_retention/purge_policy.py:
##########
@@ -905,32 +1398,283 @@ def _validated_purge_policy(model: type[Any]) -> PurgeEntityPolicy:
details = (
f"{details}; " if details else ""
) + f"stale_listeners=[{', '.join(coverage.stale_listeners)}]"
- raise RuntimeError(f"Incomplete purge policy for {model.__name__}: {details}")
+ raise RuntimeError(
+ f"Incomplete purge policy for {policy.model.__name__}: {details}"
+ )
return policy
-def _validate_executable_declarations(policy: PurgeEntityPolicy) -> None:
- """Reject executable classifications missing their required action metadata."""
+def _ownership_dependency(
+ policy: PurgeEntityPolicy, related_table: str
+) -> DependencyPolicy | None:
+ """The single owned/association edge attaching *related_table*, if clear."""
+ candidates: tuple[DependencyPolicy, ...] = tuple(
+ dependency
+ for dependency in policy.dependencies
+ if dependency.classification
+ in {DependencyClassification.OWNED, DependencyClassification.ASSOCIATION}
+ and dependency.key.related_table == related_table
+ and dependency.key.direction == "inbound"
+ )
+ return candidates[0] if len(candidates) == 1 else None
+
+
+def _validate_owned_traversal(policy: PurgeEntityPolicy) -> None:
+ """Reject an owned table reachable only through an association.
+
+ The shared cleanup empties associations before owned children, while an
+ owned table's predicate selects its rows *through* its ownership path. If
+ a hop on that path is an association, its rows are already gone when the
+ owned delete runs, so the statement matches nothing.
+
+ Kept where the rest of the ordering rules were dropped because this one
+ can be silent: where foreign keys are enforced the association delete
+ fails loudly, but where they are not -- SQLite -- the root is purged and
+ its descendants are left orphaned with nothing reported. The remedy is to
+ classify the intermediate table as owned, which places it in the same
+ phase as what it leads to.
+ """
+ root_table: str = sa.inspect(policy.model).local_table.name
for dependency in policy.dependencies:
- if (
- dependency.classification is DependencyClassification.LISTENER_EFFECT
- and dependency.listener_action is None
- ):
- raise RuntimeError(
- f"Missing listener action for {dependency.key.describe()}"
- )
- if (
- dependency.classification is DependencyClassification.VERSION_OWNED
- and dependency.version_column is None
+ if dependency.classification is not DependencyClassification.OWNED:
+ continue
+ table_name: str = dependency.key.owner_table
+ visited: set[str] = set()
+ while table_name != root_table and table_name not in visited:
+ visited.add(table_name)
+ hop: DependencyPolicy | None = _ownership_dependency(policy, table_name)
+ if hop is None:
+ # An absent or ambiguous path is reported by coverage, and by
+ # _ownership_edge at execution; not this check's business.
+ break
+ if hop.classification is DependencyClassification.ASSOCIATION:
+ raise RuntimeError(
+ f"Owned dependency {dependency.key.describe()} is reachable "
+ f"only through association {hop.key.describe()}; "
+ "associations are deleted first, so the owned rows would "
+ "be orphaned"
+ )
+ table_name = hop.key.owner_table
+
+
+def _validate_recursive_ownership(policy: PurgeEntityPolicy) -> None:
+ """Reject a self-referencing owned table under one-level cleanup.
+
+ Cleanup that issues one statement per declared edge prunes a table that
+ owns itself one level deep. Where foreign keys are enforced the root's own
+ delete then fails and rolls back; where they are not -- SQLite -- the root
+ is purged and its grandchildren are left behind with a dangling parent id
+ and nothing reported. A policy declaring a tree walks it itself and says
+ so with ``walks_own_subtree``.
+ """
+ for dependency in policy.dependencies:
+ if dependency.classification is not DependencyClassification.OWNED:
+ continue
+ key: DependencyKey = dependency.key
+ if key.owner_table != key.related_table:
+ continue
+ raise RuntimeError(
+ f"Owned dependency {key.describe()} is self-referencing; the stock "
+ "owned-child cleanup deletes one level, so this policy must supply "
+ "its own delete_owned_children"
+ )
+
+
+def _validate_scanner_requirements(policy: PurgeEntityPolicy) -> None:
+ """Reject a root the scheduled scan cannot page through.
+
+ The retention task selects, windows and orders eligible rows by ``id``,
+ and the cascade pins each row by it. A root keyed on something else -- a
+ UUID primary key with no ``id`` column -- raises inside the scan, outside
+ the per-row error handling, so the run aborts before the remaining roots
+ are reached.
+ """
+ mapper: Mapper[Any] = sa.inspect(policy.model)
+ table: sa.Table = mapper.local_table
+ if "id" not in table.c:
+ raise RuntimeError(
+ f"Purge root {policy.model.__name__} has no 'id' column; the "
+ "scheduled scan pages eligible rows by id"
+ )
+ primary_key: tuple[sa.Column[Any], ...] = tuple(mapper.primary_key)
Review Comment:
A host root with an integer `id`, `deleted_at`, and an ordinary `sa.Uuid()` `uuid` column clears this scanner validation, but `_identity_predicates` in `purge_cascade.py` then binds `uuid_column == str(entity.uuid)`. On SQLite/MySQL the non-native `Uuid` bind processor calls `.hex` on the value, so every eligible row raises a `StatementError`, stays archived, and a dry run still counts it as purgeable. Could admission also check that a `uuid` column, when present, accepts the string the identity predicate binds, or could the predicate bind the column's own Python type?
##########
tests/unit_tests/commands/deletion_retention/test_purge_policy.py:
##########
@@ -568,3 +580,1123 @@ def test_core_delete_actions_compile_for_supported_dialects(dialect: str) -> Non
assert compiled
assert all(statement.startswith(("SELECT", "DELETE")) for statement in compiled)
+
+
+#: Throwaway mapper registries created by the fixtures below, disposed after
+#: each test. Left in place they stay in SQLAlchemy's global mapper state,
+#: where ``configure_mappers()`` in the real-graph tests would walk them.
+_HOST_MAPPERS: list[registry] = []
+
+
+@pytest.fixture(autouse=True)
+def _dispose_host_mappers() -> Iterator[None]:
+ """Drop the mappers a test mapped, as the deep-path test does."""
+ yield
+ while _HOST_MAPPERS:
+ _HOST_MAPPERS.pop().dispose()
+
+
+@pytest.fixture
+def versioned_host_root(monkeypatch: pytest.MonkeyPatch) -> None:
+ """Make any root resolve a version class.
+
+ The root-relative version rules only come into play once the cascade
+ reaches the root at all, which it does by resolving the root's own version
+ class. A throwaway root has none, so these tests stand one in.
+ """
+ import sqlalchemy_continuum
+
+ monkeypatch.setattr(sqlalchemy_continuum, "version_class", lambda model: model)
+
+
+def _map_host_root(model: type[Any], table: sa.Table) -> type[Any]:
+ """Map *model* onto *table* for the duration of one test."""
+ mapper_registry: registry = registry()
+ _HOST_MAPPERS.append(mapper_registry)
+ mapper_registry.map_imperatively(model, table)
+ return model
+
+
+def _host_root(prefix: str) -> type[Any]:
+ """Map a throwaway root in its own ``MetaData``.
+
+ The hook is indifferent to a host root's shape, so this is the smallest
+ graph with anything to declare: one outbound foreign key, the same edge
+ every built-in policy preserves for its ``ab_user`` columns.
+ """
+ metadata: sa.MetaData = sa.MetaData()
+ sa.Table(
+ f"{prefix}_owner",
+ metadata,
+ sa.Column("id", sa.Integer, primary_key=True),
+ )
+ root_table: sa.Table = sa.Table(
+ f"{prefix}_entity",
+ metadata,
+ sa.Column("id", sa.Integer, primary_key=True),
+ sa.Column("owner_id", sa.Integer, sa.ForeignKey(f"{prefix}_owner.id")),
+ sa.Column("deleted_at", sa.DateTime, nullable=True),
+ )
+
+ class HostRoot:
+ """Temporary mapped root standing in for an entity core does not own."""
+
+ return _map_host_root(HostRoot, root_table)
+
+
+def _host_edge(prefix: str) -> DependencyPolicy:
+ """The one dependency ``discover_dependencies`` finds for that root."""
+ return DependencyPolicy(
+ DependencyKey(
+ "foreign_key",
+ f"{prefix}_entity",
+ f"{prefix}_owner",
+ ("owner_id",),
+ ("id",),
+ "outbound",
+ ),
+ DependencyClassification.PRESERVE,
+ )
+
+
+def _host_policy(
+ model: type[Any], dependencies: tuple[DependencyPolicy, ...]
+) -> PurgeEntityPolicy:
+ """Borrow the chart policy's actions for a host root.
+
+ The documented clone pattern, kept as the fixture so the tests exercise
+ what an integrator would actually write. The chart-specific snapshots it
+ carries are inert on a host root: each resolves from an entity type a
+ host cannot claim.
+ """
+ return replace(
+ get_purge_policy(Slice),
+ model=model,
+ entity_type="host_root",
+ dependencies=dependencies,
+ )
+
+
+@contextmanager
+def _installed(provider: Any) -> Iterator[None]:
+ """Install a host policy provider for the duration of one test.
+
+ Typed loosely on purpose: some tests install a value that is not a
+ provider at all, which is what the host boundary has to cope with.
+ """
+ previous: Any = current_app.config.get(HOST_POLICIES_CONFIG_KEY)
+ current_app.config[HOST_POLICIES_CONFIG_KEY] = provider
+ try:
+ yield
+ finally:
+ current_app.config[HOST_POLICIES_CONFIG_KEY] = previous
+
+
+def _assert_rejected(
+ model: type[Any],
+ policy: PurgeEntityPolicy,
+ reason: str,
+ caplog: pytest.LogCaptureFixture,
+) -> None:
+ """A rejected declaration costs the host its root, with the reason logged.
+
+ Rejection happens where the policy is admitted, so the root simply reads
+ as unsupported -- the shape the retention task already reports -- rather
+ than raising once per eligible row.
+ """
+ with _installed(lambda: [policy]), caplog.at_level(logging.ERROR):
+ assert model not in purge_policy_registry()
+ with pytest.raises(ValueError, match="Unsupported purge model"):
+ get_purge_policy(model)
+ assert reason in caplog.text
+
+
+def test_host_policy_extends_the_registry() -> None:
+ """A complete host declaration resolves like a built-in root."""
+ model: type[Any] = _host_root("extend")
+ policy: PurgeEntityPolicy = _host_policy(model, (_host_edge("extend"),))
+
+ with _installed(lambda: [policy]):
+ assert get_purge_policy(model) is policy
+ assert {Slice, Dashboard, SqlaTable} <= set(purge_policy_registry())
+
+
+def test_host_policy_is_held_to_the_discovered_graph(
+ caplog: pytest.LogCaptureFixture,
+) -> None:
+ """An incomplete host declaration is rejected, not silently honored."""
+ model: type[Any] = _host_root("incomplete")
+
+ _assert_rejected(model, _host_policy(model, ()), "Incomplete purge policy", caplog)
+
+
+@pytest.mark.parametrize(
+ "provider",
+ [
+ pytest.param(lambda: 1 // 0, id="raises"),
+ pytest.param(lambda: "not-a-sequence", id="wrong_type"),
+ pytest.param(lambda: [object()], id="wrong_member"),
+ ],
+)
+def test_malformed_host_provider_leaves_builtin_roots_intact(
+ provider: Callable[[], Any],
+) -> None:
+ """A broken host boundary costs the host its roots, not the purge."""
+ with _installed(provider):
+ assert set(purge_policy_registry()) == {Slice, Dashboard, SqlaTable}
+
+
+def test_host_policies_need_an_app_context() -> None:
+ """Off an app context the host's roots drop out; the built-ins remain.
+
+ The provider lives in config, so there is nothing to read without an
+ application context. Dropping the host's roots leaves them reported as
+ unsupported, where propagating the Flask error would abort the whole
+ scheduled run -- including the roots that do have policies.
+ """
+ model: type[Any] = _host_root("contextless")
+ policy: PurgeEntityPolicy = _host_policy(model, (_host_edge("contextless"),))
+
+ with (
+ _installed(lambda: [policy]),
+ patch(
+ "superset.commands.deletion_retention.purge_policy.has_app_context",
+ return_value=False,
+ ),
+ ):
+ assert set(purge_policy_registry()) == {Slice, Dashboard, SqlaTable}
+
+
+def test_duplicate_host_declarations_do_not_abort_the_registry() -> None:
+ """Two policies for one host root drop that root, not the whole index."""
+ model: type[Any] = _host_root("duplicate")
+ policy: PurgeEntityPolicy = _host_policy(model, (_host_edge("duplicate"),))
+
+ with _installed(lambda: [policy, policy]):
+ assert set(purge_policy_registry()) == {Slice, Dashboard, SqlaTable}
+ with pytest.raises(ValueError, match="Unsupported purge model"):
+ get_purge_policy(model)
+
+
+def test_host_policy_cannot_redeclare_a_builtin_root() -> None:
+ """A host cannot redefine how a chart, dashboard or dataset is purged."""
+ builtin: PurgeEntityPolicy = get_purge_policy(Slice)
+
+ with _installed(lambda: [replace(builtin, dependencies=())]):
+ assert purge_policy_registry()[Slice] is builtin
+
+
+def _host_chain(prefix: str) -> type[Any]:
+ """Map a root that reaches a detail table only through a link table."""
+ metadata: sa.MetaData = sa.MetaData()
+ root_table: sa.Table = sa.Table(
+ f"{prefix}_entity",
+ metadata,
+ sa.Column("id", sa.Integer, primary_key=True),
+ sa.Column("deleted_at", sa.DateTime, nullable=True),
+ )
+ sa.Table(
+ f"{prefix}_link",
+ metadata,
+ sa.Column("id", sa.Integer, primary_key=True),
+ sa.Column("entity_id", sa.Integer, sa.ForeignKey(f"{prefix}_entity.id")),
+ )
+ sa.Table(
+ f"{prefix}_detail",
+ metadata,
+ sa.Column("id", sa.Integer, primary_key=True),
+ sa.Column("link_id", sa.Integer, sa.ForeignKey(f"{prefix}_link.id")),
+ )
+
+ class HostChainRoot:
+ """Temporary mapped root with a two-hop ownership path."""
+
+ return _map_host_root(HostChainRoot, root_table)
+
+
+def _host_chain_edges(
+ prefix: str, link: DependencyClassification
+) -> tuple[DependencyPolicy, ...]:
+ """Classify the chain, varying only how the intermediate hop is declared."""
+ entity: str = f"{prefix}_entity"
+ link_table: str = f"{prefix}_link"
+ detail: str = f"{prefix}_detail"
+ link_phase: ExecutionPhase = (
+ ExecutionPhase.ASSOCIATIONS
+ if link is DependencyClassification.ASSOCIATION
+ else ExecutionPhase.OWNED
+ )
+ return (
+ DependencyPolicy(
+ DependencyKey(
+ "foreign_key", entity, link_table, ("id",), ("entity_id",), "inbound"
+ ),
+ link,
+ link_phase,
+ ),
+ DependencyPolicy(
+ DependencyKey(
+ "foreign_key", link_table, detail, ("id",), ("link_id",), "inbound"
+ ),
+ DependencyClassification.OWNED,
+ ExecutionPhase.OWNED,
+ ),
+ DependencyPolicy(
+ DependencyKey(
+ "foreign_key", link_table, entity, ("entity_id",), ("id",), "outbound"
+ ),
+ DependencyClassification.PRESERVE,
+ ),
+ DependencyPolicy(
+ DependencyKey(
+ "foreign_key", detail, link_table, ("link_id",), ("id",), "outbound"
+ ),
+ DependencyClassification.PRESERVE,
+ ),
+ )
+
+
+def test_owned_table_behind_an_owned_link_is_accepted() -> None:
+ """Declaring the intermediate hop owned puts both in the same phase."""
+ model: type[Any] = _host_chain("owned")
+ policy: PurgeEntityPolicy = _host_policy(
+ model, _host_chain_edges("owned", DependencyClassification.OWNED)
+ )
+
+ with _installed(lambda: [policy]):
+ assert get_purge_policy(model) is policy
+
+
+def _host_bare_root(prefix: str, *columns: str) -> type[Any]:
+ """Map a host root carrying only *columns*, to probe scan requirements."""
+ available: dict[str, sa.Column[Any]] = {
+ "id": sa.Column("id", sa.Integer, primary_key=True),
+ "uuid": sa.Column("uuid", sa.String(36), primary_key=True),
+ "deleted_at": sa.Column("deleted_at", sa.DateTime, nullable=True),
+ }
+ metadata: sa.MetaData = sa.MetaData()
+ root_table: sa.Table = sa.Table(
+ f"{prefix}_entity", metadata, *(available[name] for name in columns)
+ )
+
+ class HostBareRoot:
+ """Temporary mapped root missing a column some purge path needs."""
+
+ return _map_host_root(HostBareRoot, root_table)
+
+
+def _host_referenced(prefix: str) -> type[Any]:
+ """Map a host root with one inbound foreign key and nothing else."""
+ metadata: sa.MetaData = sa.MetaData()
+ root_table: sa.Table = sa.Table(
+ f"{prefix}_entity",
+ metadata,
+ sa.Column("id", sa.Integer, primary_key=True),
+ sa.Column("deleted_at", sa.DateTime, nullable=True),
+ )
+ sa.Table(
+ f"{prefix}_ref",
+ metadata,
+ sa.Column("id", sa.Integer, primary_key=True),
+ sa.Column("entity_id", sa.Integer, sa.ForeignKey(f"{prefix}_entity.id")),
+ )
+
+ class HostReferencedRoot:
+ """Temporary mapped root referenced by one other table."""
+
+ return _map_host_root(HostReferencedRoot, root_table)
+
+
+def _inbound_ref_key(prefix: str) -> DependencyKey:
+ return DependencyKey(
+ "foreign_key",
+ f"{prefix}_entity",
+ f"{prefix}_ref",
+ ("id",),
+ ("entity_id",),
+ "inbound",
+ )
+
+
+def _host_tree(prefix: str) -> type[Any]:
+ """Map a host root whose table owns itself through a parent column."""
+ metadata: sa.MetaData = sa.MetaData()
+ root_table: sa.Table = sa.Table(
+ f"{prefix}_node",
+ metadata,
+ sa.Column("id", sa.Integer, primary_key=True),
+ sa.Column("parent_id", sa.Integer, sa.ForeignKey(f"{prefix}_node.id")),
+ sa.Column("deleted_at", sa.DateTime, nullable=True),
+ )
+
+ class HostTreeRoot:
+ """Temporary mapped root with a recursive sub-tree."""
+
+ return _map_host_root(HostTreeRoot, root_table)
+
+
+def _host_tree_edges(prefix: str) -> tuple[DependencyPolicy, ...]:
+ node: str = f"{prefix}_node"
+ return (
+ DependencyPolicy(
+ DependencyKey(
+ "foreign_key", node, node, ("id",), ("parent_id",), "inbound"
+ ),
+ DependencyClassification.OWNED,
+ ExecutionPhase.OWNED,
+ ),
+ DependencyPolicy(
+ DependencyKey(
+ "foreign_key", node, node, ("parent_id",), ("id",), "outbound"
+ ),
+ DependencyClassification.PRESERVE,
+ ),
+ )
+
+
+def test_root_without_an_id_column_is_rejected(
+ caplog: pytest.LogCaptureFixture,
+) -> None:
+ """The scheduled scan pages by id, so a root must have one."""
+ model: type[Any] = _host_bare_root("uuidonly", "uuid")
+
+ _assert_rejected(model, _host_policy(model, ()), "has no 'id' column", caplog)
+
+
+def test_self_referencing_owned_table_accepts_declared_subtree_cleanup() -> None:
+ """A policy that says it walks the sub-tree may declare the shape."""
+ model: type[Any] = _host_tree("customtree")
+ policy: PurgeEntityPolicy = replace(
+ _host_policy(model, _host_tree_edges("customtree")),
+ delete_owned_children=lambda session, policy, entity_id: None,
+ walks_own_subtree=True,
+ )
+
+ with _installed(lambda: [policy]):
+ assert get_purge_policy(model) is policy
+
+
+def test_discovery_walks_each_recursive_table_once(
+ monkeypatch: pytest.MonkeyPatch,
+) -> None:
+ """Traversal is per table, not per permutation of tables.
+
+ Five link tables walked per permutation is already hundreds of visits;
+ a dozen is billions, which a host root could reach before any row is
+ purged.
+ """
+ metadata: sa.MetaData = sa.MetaData()
+ root_table: sa.Table = sa.Table(
+ "wide_entity",
+ metadata,
+ sa.Column("id", sa.Integer, primary_key=True),
+ )
+ link_names: list[str] = [f"wide_link{index}" for index in range(5)]
+ for name in link_names:
+ sa.Table(
+ name,
+ metadata,
+ sa.Column("id", sa.Integer, primary_key=True),
+ sa.Column("entity_id", sa.Integer, sa.ForeignKey("wide_entity.id")),
+ )
+
+ class WideRoot:
+ """Temporary mapped root owning several independent link tables."""
+
+ _map_host_root(WideRoot, root_table)
+
+ walked: list[str] = []
+ original = purge_policy_module._discover_table_dependencies # noqa: SLF001
+
+ def counting(table: sa.Table, recursive_tables: Any, seen: Any) -> Any:
+ walked.append(table.name)
+ return original(table, recursive_tables, seen)
+
+ monkeypatch.setattr(purge_policy_module, "_discover_table_dependencies", counting)
+
+ discover_dependencies(sa.inspect(WideRoot), recursive_tables=frozenset(link_names))
+
+ assert sorted(walked) == sorted(link_names)
+
+
+def test_root_without_a_deleted_at_column_is_rejected(
+ caplog: pytest.LogCaptureFixture,
+) -> None:
+ """Every purge path requires the row to be archived first."""
+ model: type[Any] = _host_bare_root("noflag", "id")
+
+ _assert_rejected(model, _host_policy(model, ()), "no 'deleted_at' column", caplog)
+
+
+def test_discovery_keeps_a_child_version_shadow_reached_through_its_mapper() -> None:
+ """A table walked plainly sees fewer edges than one walked as a mapper.
+
+ ``sql_metrics`` is reachable both ways from the dataset root. Tracking the
+ two kinds of walk together, rather than separately, dropped its version
+ shadow -- which then reads as a stale declaration on the dataset policy.
+ This pins the edge the split exists to preserve.
+ """
+ configure_mappers()
+ policy: PurgeEntityPolicy = get_purge_policy(SqlaTable)
+ recursive_tables: frozenset[str] = frozenset(
+ dependency.key.related_table
+ for dependency in policy.dependencies
+ if dependency.classification
+ in {
+ DependencyClassification.OWNED,
+ DependencyClassification.ASSOCIATION,
+ }
+ )
+
+ discovered: frozenset[DependencyKey] = discover_dependencies(
+ sa.inspect(SqlaTable), recursive_tables=recursive_tables
+ )
+
+ assert (
+ DependencyKey(
+ kind="relationship",
+ owner_table="sql_metrics",
+ related_table="sql_metrics_version",
+ direction="onetomany",
+ relationship="versions",
+ )
+ in discovered
+ )
+
+
+def test_provider_is_invoked_once_however_many_roots_resolve() -> None:
+ """The index is cached per provider, so a slow provider runs once."""
+ model: type[Any] = _host_root("once")
+ policy: PurgeEntityPolicy = _host_policy(model, (_host_edge("once"),))
+ invocations: list[int] = []
+
+ def provider() -> list[PurgeEntityPolicy]:
+ invocations.append(1)
+ return [policy]
+
+ with _installed(provider):
+ for _ in range(5):
+ assert get_purge_policy(model) is policy
+ assert get_purge_policy(Slice) is not None
+
+ assert len(invocations) == 1
+
+
+def test_host_policy_claiming_a_built_in_entity_type_is_dropped(
+ caplog: pytest.LogCaptureFixture,
+) -> None:
+ """Core branches on entity_type, so its names are reserved."""
+ model: type[Any] = _host_root("claimed")
+ policy: PurgeEntityPolicy = replace(
+ _host_policy(model, (_host_edge("claimed"),)), entity_type="dataset"
+ )
+
+ _assert_rejected(model, policy, "reserved entity type", caplog)
+
+
+@pytest.mark.parametrize(
+ "root",
+ [
+ pytest.param(
+ sa.Table(
+ "host_not_a_class",
+ sa.MetaData(),
+ sa.Column("id", sa.Integer, primary_key=True),
+ ),
+ id="table",
+ ),
+ pytest.param([], id="unhashable"),
+ ],
+)
+def test_host_policy_with_a_non_class_root_is_dropped(
+ root: Any, caplog: pytest.LogCaptureFixture
+) -> None:
+ """A root that is not a class is dropped without touching the index.
+
+ Reading or hashing it must not happen before the boundary can refuse it:
+ an exception escaping here would break the registry for the built-in
+ roots too.
+ """
+ policy: PurgeEntityPolicy = replace(
+ get_purge_policy(Slice), model=root, entity_type="host_root", dependencies=()
+ )
+
+ with _installed(lambda: [policy]), caplog.at_level(logging.ERROR):
+ assert set(purge_policy_registry()) == {Slice, Dashboard, SqlaTable}
+
+ assert "is not a class" in caplog.text
+
+
+def test_host_policies_may_arrive_as_any_iterable() -> None:
+ """The contract is a sequence of policies, not one particular container."""
+ model: type[Any] = _host_root("generated")
+ policy: PurgeEntityPolicy = _host_policy(model, (_host_edge("generated"),))
+
+ with _installed(lambda: (item for item in [policy])):
+ assert get_purge_policy(model) is policy
+
+
+def test_root_with_a_composite_primary_key_is_rejected(
+ caplog: pytest.LogCaptureFixture,
+) -> None:
+ """The scan pages by id and then fetches the row with a scalar lookup.
+
+ A composite key matches nothing, so every row fails while a dry run still
+ counts it as purgeable.
+ """
+ metadata: sa.MetaData = sa.MetaData()
+ root_table: sa.Table = sa.Table(
+ "composite_entity",
+ metadata,
+ sa.Column("tenant_id", sa.Integer, primary_key=True),
+ sa.Column("id", sa.Integer, primary_key=True),
+ sa.Column("deleted_at", sa.DateTime, nullable=True),
+ )
+
+ class CompositeRoot:
+ """Temporary mapped root keyed on two columns."""
+
+ _map_host_root(CompositeRoot, root_table)
+
+ _assert_rejected(
+ CompositeRoot, _host_policy(CompositeRoot, ()), "is keyed on", caplog
+ )
+
+
+def test_version_target_on_an_intermediate_owner_is_rejected(
+ caplog: pytest.LogCaptureFixture,
+ versioned_host_root: None,
+) -> None:
+ """Version cleanup compares the declared column with the root's id.
+
+ A column that is not root-relative matches another root's history and
+ leaves this root's behind.
+ """
+ model: type[Any] = _host_referenced("versioned")
+ declared: tuple[DependencyPolicy, ...] = (
+ DependencyPolicy(
+ DependencyKey(
+ "relationship",
+ "versioned_ref",
+ "versioned_ref_version",
+ direction="onetomany",
+ relationship="versions",
+ ),
+ DependencyClassification.VERSION_OWNED,
+ ExecutionPhase.VERSION,
+ version_column="id",
+ ),
+ )
+
+ _assert_rejected(
+ model, _host_policy(model, declared), "is not root-relative", caplog
+ )
+
+
+def test_root_whose_table_name_is_ambiguous_is_rejected(
+ caplog: pytest.LogCaptureFixture,
+) -> None:
+ """Identities are recorded by bare name while metadata keys them by schema.
+
+ With the same bare name in two schemas, cleanup would resolve to the
+ default-schema table and delete rows belonging to unrelated roots.
+ """
+ metadata: sa.MetaData = sa.MetaData()
+ sa.Table(
+ "ambiguous_entity",
+ metadata,
+ sa.Column("id", sa.Integer, primary_key=True),
+ sa.Column("deleted_at", sa.DateTime, nullable=True),
+ )
+ qualified: sa.Table = sa.Table(
+ "ambiguous_entity",
+ metadata,
+ sa.Column("id", sa.Integer, primary_key=True),
+ sa.Column("deleted_at", sa.DateTime, nullable=True),
+ schema="host",
+ )
+
+ class QualifiedRoot:
+ """Temporary mapped root living in a named schema."""
+
+ _map_host_root(QualifiedRoot, qualified)
+
+ _assert_rejected(
+ QualifiedRoot,
+ _host_policy(QualifiedRoot, ()),
+ "unambiguously by name",
+ caplog,
+ )
+
+
+def test_provider_may_build_on_a_built_in_policy() -> None:
+ """Borrowing a built-in policy inside the provider must not stall.
+
+ ``replace(get_purge_policy(Slice), ...)`` is the obvious way for a host to
+ pick up the stock callbacks, and it re-enters resolution while the first
+ call is still inside the provider. Run on a thread so a regression here
+ surfaces as a timeout rather than hanging the suite.
+ """
+ app = current_app._get_current_object() # noqa: SLF001
+ model: type[Any] = _host_root("reentrant")
+
+ def provider() -> list[PurgeEntityPolicy]:
+ # _host_policy itself resolves the chart policy, which is the
+ # re-entrant call under test.
+ return [_host_policy(model, (_host_edge("reentrant"),))]
+
+ resolved: list[PurgeEntityPolicy] = []
+
+ def resolve() -> None:
+ with app.app_context():
+ resolved.append(get_purge_policy(model))
+
+ with _installed(provider):
+ thread = threading.Thread(target=resolve)
+ thread.start()
+ thread.join(timeout=10)
+
+ assert not thread.is_alive(), "resolution did not finish: re-entry stalled"
+ assert len(resolved) == 1
+ assert resolved[0].model is model
+
+
+def test_concurrent_first_use_publishes_a_whole_index() -> None:
+ """Each caller sees a complete index, never a half-built one.
+
+ Two first uses may each invoke the provider -- the deliberate trade for
+ not holding a lock across host code -- but publishing is a single rebind,
+ so neither sees a partial result.
+ """
+ app = current_app._get_current_object() # noqa: SLF001
+ model: type[Any] = _host_root("concurrent")
+ policy: PurgeEntityPolicy = _host_policy(model, (_host_edge("concurrent"),))
+ both_ready: threading.Barrier = threading.Barrier(2)
+
+ def provider() -> list[PurgeEntityPolicy]:
+ time.sleep(0.05)
+ return [policy]
+
+ seen: list[tuple[PurgeEntityPolicy, set[type[Any]]]] = []
+
+ def resolve() -> None:
+ with app.app_context():
+ both_ready.wait(timeout=5)
+ seen.append((get_purge_policy(model), set(purge_policy_registry())))
+
+ with _installed(provider):
+ threads: list[threading.Thread] = [
+ threading.Thread(target=resolve) for _ in range(2)
+ ]
+ for thread in threads:
+ thread.start()
+ for thread in threads:
+ thread.join(timeout=10)
+
+ assert len(seen) == 2
+ for found, roots in seen:
+ assert found is policy
+ assert {Slice, Dashboard, SqlaTable} <= roots
+
+
+def test_root_with_a_non_integer_id_is_rejected(
+ caplog: pytest.LogCaptureFixture,
+) -> None:
+ """The scan pages an integer watermark, starting from zero."""
+ metadata: sa.MetaData = sa.MetaData()
+ root_table: sa.Table = sa.Table(
+ "stringid_entity",
+ metadata,
+ sa.Column("id", sa.String(36), primary_key=True),
+ sa.Column("deleted_at", sa.DateTime, nullable=True),
+ )
+
+ class StringIdRoot:
+ """Temporary mapped root keyed on a string id."""
+
+ _map_host_root(StringIdRoot, root_table)
+
+ _assert_rejected(
+ StringIdRoot, _host_policy(StringIdRoot, ()), "integer 'id'", caplog
+ )
+
+
+def test_version_target_keyed_on_a_non_primary_root_column_is_rejected(
+ caplog: pytest.LogCaptureFixture,
+ versioned_host_root: None,
+) -> None:
+ """Cleanup compares the target with the root's id, not another column.
+
+ A child keyed on some other root column matches rows whose value happens
+ to equal this root's id -- another root's history.
+ """
+ metadata: sa.MetaData = sa.MetaData()
+ root_table: sa.Table = sa.Table(
+ "coded_entity",
+ metadata,
+ sa.Column("id", sa.Integer, primary_key=True),
+ sa.Column("code", sa.Integer, unique=True),
+ sa.Column("deleted_at", sa.DateTime, nullable=True),
+ )
+ sa.Table(
+ "coded_child",
+ metadata,
+ sa.Column("id", sa.Integer, primary_key=True),
+ sa.Column("root_code", sa.Integer, sa.ForeignKey("coded_entity.code")),
+ )
+
+ class CodedRoot:
+ """Temporary mapped root whose child is keyed on a non-primary column."""
+
+ _map_host_root(CodedRoot, root_table)
+ declared: tuple[DependencyPolicy, ...] = (
+ DependencyPolicy(
+ DependencyKey(
+ "relationship",
+ "coded_child",
+ "coded_child_version",
+ direction="onetomany",
+ relationship="versions",
+ ),
+ DependencyClassification.VERSION_OWNED,
+ ExecutionPhase.VERSION,
+ version_column="root_code",
+ ),
+ )
+
+ _assert_rejected(
+ CodedRoot, _host_policy(CodedRoot, declared), "is not root-relative", caplog
+ )
+
+
+@pytest.mark.parametrize(
+ "classification",
+ [
+ pytest.param(DependencyClassification.BLOCK, id="block"),
+ pytest.param(DependencyClassification.LISTENER_EFFECT, id="listener_effect"),
+ ],
+)
+def test_host_policy_declaring_core_only_dependencies_is_dropped(
+ classification: DependencyClassification, caplog: pytest.LogCaptureFixture
+) -> None:
+ """Both resolve from core's own identities, so a host declaration is silent.
+
+ The stock listener actions decide what to delete from a built-in entity
+ type, and blockers are applied by the stock validator. Declared by a host
+ either one is accepted and then never carried out -- the one outcome worth
+ refusing, since nothing announces it. A host refuses a purge by raising
+ PurgeBlockedError from its own validator instead.
+ """
+ model: type[Any] = _host_referenced("coreonly")
+ declared: tuple[DependencyPolicy, ...] = (
+ DependencyPolicy(
+ _inbound_ref_key("coreonly"),
+ classification,
+ ExecutionPhase.VALIDATE,
+ blocker=BlockerReason("host_reference", "a reference exists"),
+ listener_action=ListenerAction.DELETE_TAGGED_OBJECTS,
+ ),
+ )
+
+ _assert_rejected(
+ model, _host_policy(model, declared), "not available to a host root", caplog
+ )
+
+
+def test_owned_table_behind_an_association_is_rejected(
+ caplog: pytest.LogCaptureFixture,
+) -> None:
+ """Associations are emptied first, so owning through one cannot execute.
+
+ Kept where the other ordering rules were dropped because this one can be
+ silent: without enforced foreign keys the root purges and its descendants
+ are orphaned with nothing reported.
+ """
+ model: type[Any] = _host_chain("behind")
+ policy: PurgeEntityPolicy = _host_policy(
+ model, _host_chain_edges("behind", DependencyClassification.ASSOCIATION)
+ )
+
+ _assert_rejected(model, policy, "associations are deleted first", caplog)
+
+
+def test_a_failing_provider_is_not_called_again_on_every_read() -> None:
+ """A failure answers reads for a while instead of re-running the provider.
+
+ The registry is read roughly three times per purged entity, so retrying
+ per read meant tens of thousands of provider calls -- and tracebacks --
+ in a single pass.
+ """
+ calls: list[int] = []
+
+ def provider() -> list[PurgeEntityPolicy]:
+ calls.append(1)
+ raise RuntimeError("manager unreachable")
+
+ with _installed(provider):
+ for _ in range(4):
+ assert set(purge_policy_registry()) == {Slice, Dashboard, SqlaTable}
+
+ assert len(calls) == 1
+
+
+def test_a_failing_provider_is_tried_again_once_its_window_passes(
+ monkeypatch: pytest.MonkeyPatch,
+) -> None:
+ """And not remembered for the life of the process either.
+
+ The window is set to zero here so the published failure is already stale,
+ which is what a later purge sees after a transient outage.
+ """
+ monkeypatch.setattr(purge_policy_module, "_PROVIDER_RETRY_SECONDS", 0.0)
+ model: type[Any] = _host_root("transient")
+ policy: PurgeEntityPolicy = _host_policy(model, (_host_edge("transient"),))
+ calls: list[int] = []
+
+ def provider() -> list[PurgeEntityPolicy]:
+ calls.append(1)
+ if len(calls) == 1:
+ raise RuntimeError("manager unreachable")
+ return [policy]
+
+ with _installed(provider):
+ assert model not in purge_policy_registry()
+ assert get_purge_policy(model) is policy
+
+ assert len(calls) == 2
+
+
+def test_replacing_only_the_association_delete_still_refuses_the_shape(
+ caplog: pytest.LogCaptureFixture,
+) -> None:
+ """The orphaning hazard belongs to the stock owned cleanup alone.
+
+ Whoever empties the association rows, the stock owned delete still
+ traverses an ownership path that is empty by the time it runs.
+ """
+ model: type[Any] = _host_chain("assoconly")
+ policy: PurgeEntityPolicy = replace(
+ _host_policy(
+ model, _host_chain_edges("assoconly", DependencyClassification.ASSOCIATION)
+ ),
+ delete_associations=lambda session, policy, entity_id: None,
+ )
+
+ _assert_rejected(model, policy, "associations are deleted first", caplog)
+
+
+def test_self_referencing_owned_table_rejects_stock_cleanup(
+ caplog: pytest.LogCaptureFixture,
+) -> None:
+ """Stock cleanup prunes one level, which orphans a deeper tree.
+
+ Loud where foreign keys are enforced, silent on SQLite: the root is
+ purged and its grandchildren keep a dangling parent id.
+ """
+ model: type[Any] = _host_tree("stocktree")
+ policy: PurgeEntityPolicy = _host_policy(model, _host_tree_edges("stocktree"))
+
+ _assert_rejected(model, policy, "must supply its own", caplog)
+
+
+def test_self_referencing_version_target_is_rejected(
+ caplog: pytest.LogCaptureFixture,
+ versioned_host_root: None,
+) -> None:
+ """A parent column on the root's own shadow is not the root's identity.
+
+ Cleanup compares it with the purged row's id, so it deletes that row's
+ children's history and leaves its own behind.
+ """
+ model: type[Any] = _host_tree("selfversion")
+ declared: tuple[DependencyPolicy, ...] = (
+ *_host_tree_edges("selfversion"),
+ DependencyPolicy(
+ DependencyKey(
+ "relationship",
+ "selfversion_node",
+ "selfversion_node_version",
+ direction="onetomany",
+ relationship="versions",
+ ),
+ DependencyClassification.VERSION_OWNED,
+ ExecutionPhase.VERSION,
+ version_column="parent_id",
+ ),
+ )
+ policy: PurgeEntityPolicy = replace(
+ _host_policy(model, declared),
+ delete_owned_children=lambda session, policy, entity_id: None,
+ )
+
+ _assert_rejected(model, policy, "is not root-relative", caplog)
+
+
+def test_each_app_resolves_its_own_index(app_context: None) -> None:
+ """Two apps in one process do not thrash a single global snapshot.
+
+ Without per-app state each read from either app would see the other's
+ provider, re-resolve, and call the provider again -- once per root.
+ """
+ from superset.app import SupersetApp
+
+ first = current_app._get_current_object() # noqa: SLF001
+ model: type[Any] = _host_root("perapp")
+ policy: PurgeEntityPolicy = _host_policy(model, (_host_edge("perapp"),))
+ calls: list[int] = []
+
+ def provider() -> list[PurgeEntityPolicy]:
+ calls.append(1)
+ return [policy]
+
+ second = SupersetApp(__name__)
+ second.config.update(first.config)
+ second.config[HOST_POLICIES_CONFIG_KEY] = None
+
+ with _installed(provider):
+ assert get_purge_policy(model) is policy
+ with second.app_context():
+ # The second app has no provider, so its index carries only the
+ # built-in roots -- and resolving it must not disturb the first.
+ assert model not in purge_policy_registry()
+ assert get_purge_policy(model) is policy
+
+ assert len(calls) == 1
+
+
+def test_a_lazy_payload_that_raises_is_retried_like_the_call(
+ monkeypatch: pytest.MonkeyPatch,
+) -> None:
+ """Walking a generator runs host code, so it earns the provider's retry.
+
+ Reading a lazy payload can fail for the same transient reasons calling
+ the provider can; only the shape of a payload that arrived intact is
+ settled for good.
+ """
+ monkeypatch.setattr(purge_policy_module, "_PROVIDER_RETRY_SECONDS", 0.0)
+ model: type[Any] = _host_root("lazy")
+ policy: PurgeEntityPolicy = _host_policy(model, (_host_edge("lazy"),))
+ calls: list[int] = []
+
+ def provider() -> Iterator[PurgeEntityPolicy]:
+ calls.append(1)
+ if len(calls) == 1:
+ raise RuntimeError("manager unreachable")
+ yield policy # pragma: no cover - unreachable, keeps this a generator
+ yield policy
+
+ with _installed(provider):
+ assert model not in purge_policy_registry()
+ assert get_purge_policy(model) is policy
+
+ assert len(calls) == 2
+
+
+def test_version_target_on_an_unversioned_root_is_rejected(
+ caplog: pytest.LogCaptureFixture,
+) -> None:
+ """The cascade stops before a declared target when the root has none.
+
+ It resolves the root's own version class first, so for an unversioned root
+ a target on a child's shadow is never reached: the root and its live child
+ rows are purged while the child's history survives, unreported.
+ """
+ model: type[Any] = _host_referenced("unversioned")
+ declared: tuple[DependencyPolicy, ...] = (
+ DependencyPolicy(
+ DependencyKey(
+ "relationship",
+ "unversioned_ref",
+ "unversioned_ref_version",
+ direction="onetomany",
+ relationship="versions",
+ ),
+ DependencyClassification.VERSION_OWNED,
+ ExecutionPhase.VERSION,
+ version_column="entity_id",
+ ),
+ )
+
+ _assert_rejected(
+ model, _host_policy(model, declared), "has no version class", caplog
+ )
+
+
+def test_host_policy_mapped_onto_a_built_in_table_is_dropped(
+ caplog: pytest.LogCaptureFixture,
+) -> None:
+ """A different class on a built-in table is still this package's rows.
+
+ It is not the built-in root by identity, so the class check alone admits
+ it -- and then host cleanup runs against ``slices``. Inheriting the mixin
+ also puts it in the soft-delete registry, so the scheduled task would
+ scan it.
+ """
+
+ class HostSliceClone(Slice):
Review Comment:
Defining `HostSliceClone(Slice)` here appends it to the real `SoftDeleteMixin._registered_subclasses` through `__init_subclass__`, and neither this test nor `_dispose_host_mappers` removes it afterward. Any later test in the same process that scans the live registry will then see a second `slices` model and report `unsupported_models == {"slices": 1}` even though the chart policy is installed. Could this snapshot and restore the registry list the way the `NoTableName` test does?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(retention): let a host install purge policies for its own soft-delete roots [superset]
Posted by "msyavuz (via GitHub)" <gi...@apache.org>.
msyavuz commented on code in PR #44892:
URL: https://github.com/apache/superset/pull/44892#discussion_r4206637630
##########
superset/commands/deletion_retention/purge_policy.py:
##########
@@ -860,16 +923,444 @@ def declare(
cleanup_permission=cleanup_dataset_permission,
),
}
- return validate_unique_root_policies(registry.values())
+ return tuple(registry.values())
+
+def _policy_label(policy: Any) -> str:
+ """A log-safe name for a payload whose shape is not yet established."""
+ model: Any = getattr(policy, "model", None)
+ return str(getattr(model, "__name__", type(model).__name__))
+
+
+def _host_policy_payload(provided: Any) -> tuple[Any, ...]:
+ """Materialize the provider's payload, or reject its shape.
+
+ Any iterable is accepted -- a list, a tuple, ``dict.values()``, a
+ generator -- since the documented contract is a sequence of policies, not
+ one particular container. Strings and bytes are excluded because they
+ iterate into characters, which would read as a sequence of bad members
+ rather than the wrong type. Materialized once: a generator cannot be
+ walked twice.
+
+ An exception raised *while reading* a lazy payload is left to propagate.
+ Walking a generator runs host code just as calling the provider does, so
+ the two are the same kind of failure and earn the same retry; only the
+ shape of a payload that arrived intact is settled for good.
+ """
+ if isinstance(provided, (str, bytes)) or not isinstance(provided, Iterable):
+ logger.error(
+ "purge_policy: %s returned %s; expected an iterable of PurgeEntityPolicy",
+ HOST_POLICIES_CONFIG_KEY,
+ type(provided).__name__,
+ )
+ return ()
+ return tuple(provided)
-@lru_cache(maxsize=None)
-def _validated_purge_policy(model: type[Any]) -> PurgeEntityPolicy:
- """Validate and return one root policy without blocking unrelated roots."""
+
+def _unqualified_table_conflict(policy: PurgeEntityPolicy) -> str | None:
+ """Return the first table a bare-name lookup would resolve wrongly.
+
+ Discovery records table identities by bare name while a ``MetaData`` keys
+ them by schema, so a declared ``child`` resolves to the default-schema
+ table even where the policy meant ``host.child`` -- and the cleanup would
+ delete rows belonging to unrelated roots. Carrying schema-qualified
+ identities through discovery and execution is the better fix; until then
+ the ambiguous shape is refused rather than silently mis-resolved.
+ """
+ root_table: sa.Table = sa.inspect(policy.model).local_table
+ metadata: sa.MetaData = root_table.metadata
+ keys_by_name: dict[str, list[str]] = {}
+ for table in metadata.tables.values():
+ keys_by_name.setdefault(table.name, []).append(table.key)
+ declared: set[str] = {root_table.name} | {
+ dependency.key.related_table
+ for dependency in policy.dependencies
+ # Version shadows are resolved by bare name as well, in
+ # _entity_version_targets.
+ if dependency.classification in _EXECUTABLE_CLASSIFICATIONS
+ or dependency.classification is DependencyClassification.VERSION_OWNED
+ }
+ for name in sorted(declared):
+ keys: list[str] = keys_by_name.get(name, [])
+ if len(keys) != 1 or keys[0] != name:
+ return name
+ return None
+
+
+def _admitted_host_policy(
+ candidate: Any,
+ builtin_roots: frozenset[type[Any]],
+ builtin_tables: frozenset[str],
+ builtin_types: frozenset[str],
+) -> PurgeEntityPolicy | None:
+ """Return *candidate* if it is a usable host policy, else ``None``.
+
+ Every check runs behind one handler rather than guarding each attribute:
+ the payload is host-supplied, so reading ``model`` or hashing it may
+ itself raise -- and an exception escaping here would break
+ ``purge_policy_registry`` for the built-in roots too, which is the
+ opposite of what this boundary is for.
+ """
try:
- policy: PurgeEntityPolicy = purge_policy_registry()[model]
- except KeyError as ex:
- raise ValueError(f"Unsupported purge model: {model.__name__}") from ex
+ if not isinstance(candidate, PurgeEntityPolicy):
+ logger.error(
+ "purge_policy: %s returned a %s; expected PurgeEntityPolicy",
+ HOST_POLICIES_CONFIG_KEY,
+ type(candidate).__name__,
+ )
+ return None
+ if not isinstance(candidate.model, type):
+ # Everything downstream treats the root as a mapped class:
+ # sa.inspect, the registry index, the duplicate count.
+ logger.error(
+ "purge_policy: host policy root %s is not a class",
+ _policy_label(candidate),
+ )
+ return None
+ if candidate.model in builtin_roots:
+ logger.error(
+ "purge_policy: host policy for built-in root %s ignored",
+ candidate.model.__name__,
+ )
+ return None
+ root_table: sa.Table = sa.inspect(candidate.model).local_table
+ if root_table.key in builtin_tables:
+ # A different class mapped onto a built-in table -- a subclass of
+ # Slice, say -- is not the built-in root by identity, but purging
+ # it runs host cleanup against this package's own rows. It also
+ # joins the soft-delete registry through the mixin, so the
+ # scheduled task would scan it.
+ logger.error(
+ "purge_policy: host policy root %s is mapped onto built-in "
+ "table %r; ignored",
+ candidate.model.__name__,
+ root_table.key,
+ )
+ return None
+ if not isinstance(candidate.entity_type, str) or not candidate.entity_type:
+ # Read as a string by the cascade, the audit record and every log
+ # line that names the root.
+ logger.error(
+ "purge_policy: host policy for %s declares entity_type %r",
+ candidate.model.__name__,
+ candidate.entity_type,
+ )
+ return None
+ if candidate.entity_type in builtin_types:
+ # Core branches on entity_type -- the dataset impact path, the tag
+ # object type, the dataset permission name -- so a host reusing
+ # one of its names would have that logic pointed at its own rows.
+ logger.error(
+ "purge_policy: host policy for %s claims the reserved entity type %r",
+ candidate.model.__name__,
+ candidate.entity_type,
+ )
+ return None
+ # Validated first, so a table the metadata simply does not contain is
+ # reported as such rather than as an ambiguous name.
+ unusable: set[str] = {
+ dependency.classification.value
+ for dependency in candidate.dependencies
+ if dependency.classification
+ in {
+ DependencyClassification.BLOCK,
+ DependencyClassification.LISTENER_EFFECT,
+ }
+ }
+ if unusable:
+ # Two different reasons, both about identities this package owns.
+ # A stock listener action decides what to delete from a built-in
+ # entity type, so declared by a host it is accepted and then never
+ # runs. A blocker *would* be applied -- validate_deletion_allowed
+ # is generic over the declarations -- but its reason code lands in
+ # the purge audit record, whose vocabulary is a closed set pinned
+ # by test. A host refuses a purge from its own validator instead,
+ # where the code it records is plainly its own.
+ raise RuntimeError(
+ f"{', '.join(sorted(unusable))} dependencies are not available "
+ "to a host root"
+ )
+ _validated_policy(candidate)
+ conflict: str | None = _unqualified_table_conflict(candidate)
+ if conflict is not None:
+ raise RuntimeError(
+ f"table {conflict!r} cannot be identified unambiguously by "
+ "name; schema-qualified roots are not supported"
+ )
+ except Exception as ex: # pylint: disable=broad-except
+ # Validated at admission rather than when a row is purged: a
+ # declaration the cleanup cannot execute would otherwise fail once per
+ # eligible row, counted as a cascade failure, and stay invisible until
+ # something aged past the window.
+ logger.error(
+ "purge_policy: host policy for %s rejected: %s",
+ _policy_label(candidate),
+ ex,
+ )
+ return None
+ return candidate
+
+
+def _host_purge_policies(
+ provider: Callable[[], Any] | None,
+) -> tuple[PurgeEntityPolicy, ...] | None:
+ """Return the purge policies a host installed for its own roots.
+
+ A host distribution can carry ``SoftDeleteMixin`` entities this package
+ cannot import. The retention task discovers those entities through the
+ mixin registry, so without a policy they reach the cascade as an
+ unsupported model. The host therefore declares their purge behavior and
+ installs it under ``PURGE_POLICIES_FUNC``.
+
+ Host boundary: everything about a host policy is settled here, where it is
+ admitted -- an unavailable provider, a payload that is not an iterable of
+ policies, a root that is not a class, a collision with a root or entity
+ type declared in this package, two declarations for one root, and a
+ declaration the shared cleanup could not execute. Each is logged and
+ dropped, so that root is reported as unsupported instead of failing row by
+ row in a scheduled run. A broken host declaration must not take the purge
+ down with it, and must never redefine how a chart, dashboard or dataset is
+ purged.
+
+ Returns ``None`` for the failures that may be transient -- the provider
+ could not be called, or a lazy payload raised while being read. The caller
+ then publishes the built-in roots with a short deadline, so the next
+ resolution after it tries again rather than treating one bad moment as the
+ answer for the life of the process. A payload that arrived intact but is
+ the wrong shape is a host bug that will not fix itself, so it is published
+ as an empty result instead.
+ """
+ if provider is None:
+ return ()
+ if not callable(provider):
+ # Static misconfiguration, not an outage: reported once rather than
+ # retried with a traceback for the life of the process.
+ logger.error(
+ "purge_policy: %s is %s, not a callable returning policies",
+ HOST_POLICIES_CONFIG_KEY,
+ type(provider).__name__,
+ )
+ return ()
+ try:
+ # Both the call and reading what it returns run host code, so a lazy
+ # payload that raises mid-walk is the same kind of failure as a
+ # provider that raises outright, and earns the same retry.
+ candidates: tuple[Any, ...] = _host_policy_payload(provider())
+ except Exception: # pylint: disable=broad-except
+ logger.exception(
+ "purge_policy: %s is unavailable; keeping built-in roots only",
+ HOST_POLICIES_CONFIG_KEY,
+ )
+ return None
+
+ builtin_policies: tuple[PurgeEntityPolicy, ...] = _builtin_purge_policies()
+ builtin_roots: frozenset[type[Any]] = frozenset(
+ policy.model for policy in builtin_policies
+ )
+ builtin_tables: frozenset[str] = frozenset(
+ sa.inspect(policy.model).local_table.key for policy in builtin_policies
+ )
+ builtin_types: frozenset[str] = frozenset(
+ policy.entity_type for policy in builtin_policies
+ )
+ admitted: list[PurgeEntityPolicy] = [
+ policy
+ for candidate in candidates
+ if (
+ policy := _admitted_host_policy(
+ candidate, builtin_roots, builtin_tables, builtin_types
+ )
+ )
+ is not None
+ ]
+
+ # Counted after admission, where every root is known to be a class and so
+ # hashable. Which of two declarations for one root is authoritative is
+ # undecidable, and the loser would still delete rows, so both are dropped:
+ # the root is reported as unsupported, which is the recoverable outcome.
+ # It also keeps the duplicate away from validate_unique_root_policies,
+ # whose ValueError would abort the whole scheduled run.
+ counts: dict[type[Any], int] = {}
+ for policy in admitted:
+ counts[policy.model] = counts.get(policy.model, 0) + 1
+ for model, count in counts.items():
+ if count > 1:
+ logger.error(
+ "purge_policy: %s declares %d policies for %s; ignoring all of them",
+ HOST_POLICIES_CONFIG_KEY,
+ count,
+ model.__name__,
+ )
+ return tuple(policy for policy in admitted if counts[policy.model] == 1)
+
+
+#: Sentinel distinguishing "no index resolved yet" from an index resolved for
+#: no provider, which is the ordinary case.
+_UNRESOLVED: Any = object()
+
+
+@dataclass
+class _ResolvedRegistry:
+ """The index built for one provider, with the roots validated so far.
+
+ Matched to its provider by *identity* rather than keyed in an
+ ``lru_cache``: a cache key would have to be hashed, and a host callable
+ implemented as a mutable dataclass instance is unhashable. That raises
+ before the host boundary can isolate the host's mistake, which would
+ stop the built-in roots purging too.
+ """
+
+ provider: Any = _UNRESOLVED
+ registry: Mapping[type[Any], PurgeEntityPolicy] = field(
+ default_factory=lambda: MappingProxyType({})
+ )
+ validated: set[type[Any]] = field(default_factory=set)
+ #: Set only on a snapshot published because the provider could not be
+ #: called. Until this deadline the snapshot answers reads; after it, the
+ #: provider is tried again.
+ retry_after: float | None = None
+
+
+#: Key the resolved index lives under on the Flask app. Per app rather than
+#: per process: two apps in one process have different configs, and a single
+#: global snapshot would thrash between them -- re-resolving, and re-invoking
+#: the provider, on every read.
+_REGISTRY_EXTENSION_KEY: str = "deletion_retention_purge_registry"
+
+#: How long a snapshot published after a provider failure answers reads
+#: before the provider is tried again. Bounds both extremes: a failure is
+#: neither remembered for the life of the process nor retried on every read --
+#: at roughly three reads per purged entity, retrying per read meant tens of
+#: thousands of provider calls and tracebacks in a single pass.
+_PROVIDER_RETRY_SECONDS: float = 60.0
+
+#: Per-thread marker for "this thread is already resolving". A host provider
+#: may build its policy from a built-in one -- ``replace(get_purge_policy(
+#: Slice), model=HostRoot, ...)`` is the obvious way to borrow the stock
+#: callbacks -- which re-enters resolution while the first call is still in
+#: the provider. The nested call is handed the state already published
+#: instead, which always carries the built-in roots.
+_RESOLVING: threading.local = threading.local()
+
+
+def _resolved_registry() -> _ResolvedRegistry:
+ """Return the state resolved for the installed provider, building it once.
+
+ Readers take a snapshot and a rebuild publishes a new one, so a caller
+ always sees an index and its validation record together. Publishing is a
+ single rebind, which is what makes it safe to do this without a lock: two
+ concurrent first uses may each invoke the provider and each publish, but
+ the two snapshots are equivalent and neither is ever seen half-built.
+ Paying for a duplicate provider call is the deliberate trade -- a lock
+ here deadlocks the moment a provider resolves a built-in policy of its
+ own, and a hung purge is worse than a repeated call.
+ """
+ if not has_app_context():
+ return _builtin_registry()
+ provider: Callable[[], Any] | None = current_app.config.get(
+ HOST_POLICIES_CONFIG_KEY
+ )
+ published: _ResolvedRegistry | None = current_app.extensions.get(
+ _REGISTRY_EXTENSION_KEY
+ )
+ if published is None:
+ published = _builtin_registry()
+ current_app.extensions[_REGISTRY_EXTENSION_KEY] = published
+ if published.provider is provider and (
+ published.retry_after is None or time.monotonic() < published.retry_after
+ ):
+ return published
+ if getattr(_RESOLVING, "active", False):
Review Comment:
`_RESOLVING.active` is only checked after the cache-hit return, so a nested resolution that finds a stale or failed snapshot returns `_builtin_registry()`; if the provider calls `purge_policy_registry()` and expects host roots, it silently gets built-ins only.
##########
superset/commands/deletion_retention/purge_policy.py:
##########
@@ -905,32 +1398,283 @@ def _validated_purge_policy(model: type[Any]) -> PurgeEntityPolicy:
details = (
f"{details}; " if details else ""
) + f"stale_listeners=[{', '.join(coverage.stale_listeners)}]"
- raise RuntimeError(f"Incomplete purge policy for {model.__name__}: {details}")
+ raise RuntimeError(
+ f"Incomplete purge policy for {policy.model.__name__}: {details}"
+ )
return policy
-def _validate_executable_declarations(policy: PurgeEntityPolicy) -> None:
- """Reject executable classifications missing their required action metadata."""
+def _ownership_dependency(
+ policy: PurgeEntityPolicy, related_table: str
+) -> DependencyPolicy | None:
+ """The single owned/association edge attaching *related_table*, if clear."""
+ candidates: tuple[DependencyPolicy, ...] = tuple(
+ dependency
+ for dependency in policy.dependencies
+ if dependency.classification
+ in {DependencyClassification.OWNED, DependencyClassification.ASSOCIATION}
+ and dependency.key.related_table == related_table
+ and dependency.key.direction == "inbound"
+ )
+ return candidates[0] if len(candidates) == 1 else None
+
+
+def _validate_owned_traversal(policy: PurgeEntityPolicy) -> None:
+ """Reject an owned table reachable only through an association.
+
+ The shared cleanup empties associations before owned children, while an
+ owned table's predicate selects its rows *through* its ownership path. If
+ a hop on that path is an association, its rows are already gone when the
+ owned delete runs, so the statement matches nothing.
+
+ Kept where the rest of the ordering rules were dropped because this one
+ can be silent: where foreign keys are enforced the association delete
+ fails loudly, but where they are not -- SQLite -- the root is purged and
+ its descendants are left orphaned with nothing reported. The remedy is to
+ classify the intermediate table as owned, which places it in the same
+ phase as what it leads to.
+ """
+ root_table: str = sa.inspect(policy.model).local_table.name
for dependency in policy.dependencies:
- if (
- dependency.classification is DependencyClassification.LISTENER_EFFECT
- and dependency.listener_action is None
- ):
- raise RuntimeError(
- f"Missing listener action for {dependency.key.describe()}"
- )
- if (
- dependency.classification is DependencyClassification.VERSION_OWNED
- and dependency.version_column is None
+ if dependency.classification is not DependencyClassification.OWNED:
+ continue
+ table_name: str = dependency.key.owner_table
+ visited: set[str] = set()
+ while table_name != root_table and table_name not in visited:
+ visited.add(table_name)
+ hop: DependencyPolicy | None = _ownership_dependency(policy, table_name)
+ if hop is None:
+ # An absent or ambiguous path is reported by coverage, and by
+ # _ownership_edge at execution; not this check's business.
+ break
+ if hop.classification is DependencyClassification.ASSOCIATION:
+ raise RuntimeError(
+ f"Owned dependency {dependency.key.describe()} is reachable "
+ f"only through association {hop.key.describe()}; "
+ "associations are deleted first, so the owned rows would "
+ "be orphaned"
+ )
+ table_name = hop.key.owner_table
+
+
+def _validate_recursive_ownership(policy: PurgeEntityPolicy) -> None:
+ """Reject a self-referencing owned table under one-level cleanup.
+
+ Cleanup that issues one statement per declared edge prunes a table that
+ owns itself one level deep. Where foreign keys are enforced the root's own
+ delete then fails and rolls back; where they are not -- SQLite -- the root
+ is purged and its grandchildren are left behind with a dangling parent id
+ and nothing reported. A policy declaring a tree walks it itself and says
+ so with ``walks_own_subtree``.
+ """
+ for dependency in policy.dependencies:
+ if dependency.classification is not DependencyClassification.OWNED:
+ continue
+ key: DependencyKey = dependency.key
+ if key.owner_table != key.related_table:
+ continue
+ raise RuntimeError(
+ f"Owned dependency {key.describe()} is self-referencing; the stock "
+ "owned-child cleanup deletes one level, so this policy must supply "
+ "its own delete_owned_children"
+ )
+
+
+def _validate_scanner_requirements(policy: PurgeEntityPolicy) -> None:
+ """Reject a root the scheduled scan cannot page through.
+
+ The retention task selects, windows and orders eligible rows by ``id``,
+ and the cascade pins each row by it. A root keyed on something else -- a
+ UUID primary key with no ``id`` column -- raises inside the scan, outside
+ the per-row error handling, so the run aborts before the remaining roots
+ are reached.
+ """
+ mapper: Mapper[Any] = sa.inspect(policy.model)
+ table: sa.Table = mapper.local_table
+ if "id" not in table.c:
+ raise RuntimeError(
+ f"Purge root {policy.model.__name__} has no 'id' column; the "
+ "scheduled scan pages eligible rows by id"
+ )
+ primary_key: tuple[sa.Column[Any], ...] = tuple(mapper.primary_key)
+ if (
+ len(primary_key) != 1
+ or primary_key[0].name != "id"
+ or not isinstance(primary_key[0].type, sa.Integer)
+ ):
+ # The scan pages by id and the row is then fetched with a scalar
+ # Session.get, so a composite or differently named key matches
+ # nothing: every row fails while a dry run still counts it eligible.
+ raise RuntimeError(
+ f"Purge root {policy.model.__name__} is keyed on "
+ f"({', '.join(column.name for column in primary_key)}); the "
+ "scheduled purge pages an integer 'id' watermark and looks rows "
+ "up by it"
+ )
+ if "deleted_at" not in table.c:
+ raise RuntimeError(
+ f"Purge root {policy.model.__name__} has no 'deleted_at' column; "
+ "every purge path requires the row to be archived first"
+ )
+
+
+#: Classifications the shared cleanup executes as SQL, through
+#: ``_dependency_predicates`` -- which only knows how to read an inbound
+#: foreign key.
+_EXECUTABLE_CLASSIFICATIONS: frozenset[DependencyClassification] = frozenset(
+ {
+ DependencyClassification.OWNED,
+ DependencyClassification.ASSOCIATION,
+ DependencyClassification.BLOCK,
+ }
+)
+
+
+def _validate_versioned_root(policy: PurgeEntityPolicy) -> None:
+ """Reject version targets on a root that has no version class.
+
+ ``_delete_version_history`` resolves the *root's* version class first and
+ returns as soon as that raises, so a target declared on a child's shadow
+ is never reached for an unversioned root: the root and its live child rows
+ are purged while the child's history survives, with nothing reported.
+
+ Skipped where nothing is versioned at all. The version cascade is then
+ inert for every root, including this package's own, and refusing would
+ only take those down with it.
+ """
+ try:
+ from sqlalchemy_continuum import version_class, versioning_manager
+ from sqlalchemy_continuum.exc import ClassNotVersioned
+ except ImportError: # pragma: no cover - versioning not installed
+ return
+ if not getattr(versioning_manager, "version_class_map", None):
+ return
+ try:
+ version_class(policy.model)
+ except ClassNotVersioned:
+ raise RuntimeError(
+ f"Purge root {policy.model.__name__} has no version class, so the "
+ "version cascade returns before reaching a declared target; such "
+ "a policy declares none"
+ ) from None
+
+
+def _validate_version_target(
+ policy: PurgeEntityPolicy, dependency: DependencyPolicy, metadata: sa.MetaData
+) -> None:
+ """Reject a version target whose column does not hold the root's id.
+
+ Version cleanup compares the declared column with the root's id directly,
+ so the column has to be root-relative. Shadow tables are generated and may
+ not exist when a policy is validated, so the check reads their live
+ counterpart -- ``<table>_version`` describes ``<table>``, the naming the
+ shadow declarations already rely on -- and accepts the column when it is
+ the root's own primary key or a foreign key into the root.
+
+ A column naming an intermediate owner, a stage id under a workflow root
+ say, matches history belonging to a different root and leaves this one's
+ behind.
+ """
+ root_table: sa.Table = sa.inspect(policy.model).local_table
+ column: str = cast(str, dependency.version_column)
+ shadow_name: str = dependency.key.related_table
+ suffix: str = "_version"
+ live_name: str = (
+ shadow_name[: -len(suffix)] if shadow_name.endswith(suffix) else shadow_name
+ )
+ live_table: sa.Table | None = metadata.tables.get(live_name)
+ if live_table is not None and column in live_table.c:
+ live_column: sa.Column[Any] = live_table.c[column]
+ primary_key: tuple[sa.Column[Any], ...] = tuple(root_table.primary_key.columns)
+ if live_table is root_table:
+ # On the root's own shadow only its key identifies it. A
+ # self-referencing column such as ``parent_id`` points at *other*
+ # rows of the same root, so cleanup would delete the children's
+ # history and leave the purged row's own behind.
+ if len(primary_key) == 1 and live_column.name == primary_key[0].name:
+ return
+ # Elsewhere the comparison is still against the root's id, so a
+ # foreign key to any other root column matches rows belonging to a
+ # different root.
+ elif len(primary_key) == 1 and any(
+ foreign_key.column.table is root_table
+ and foreign_key.column.name == primary_key[0].name
+ for foreign_key in live_column.foreign_keys
):
- raise RuntimeError(
- f"Missing version target column for {dependency.key.describe()}"
- )
+ return
+ raise RuntimeError(
+ f"Version target {shadow_name}.{column} for {dependency.key.describe()} "
+ f"is not root-relative; cleanup compares it with the "
+ f"{root_table.name} id, so it must be that id or a foreign key to it"
+ )
+
+
+def _validate_dependency_declaration(
+ policy: PurgeEntityPolicy,
+ dependency: DependencyPolicy,
+ metadata: sa.MetaData,
+) -> None:
+ """Reject one declaration whose mistake would not announce itself."""
+ key: DependencyKey = dependency.key
+ if (
+ dependency.classification is DependencyClassification.LISTENER_EFFECT
+ and dependency.listener_action is None
+ ):
+ raise RuntimeError(f"Missing listener action for {key.describe()}")
+ if dependency.classification is DependencyClassification.VERSION_OWNED:
+ if dependency.version_column is None:
+ raise RuntimeError(f"Missing version target column for {key.describe()}")
+ # Both silent either way: a target the version cascade never
+ # reaches, and one that is not root-relative and so deletes another
+ # root's history while leaving this root's behind.
+ _validate_versioned_root(policy)
+ _validate_version_target(policy, dependency, metadata)
+
+
+def _validate_executable_declarations(policy: PurgeEntityPolicy) -> None:
+ """Reject a declaration that would delete the wrong rows, and little else.
+
+ Validation here is deliberately not an attempt to prove that an arbitrary
+ declaration executes correctly -- the space of shapes a host might write
+ is not enumerable, and every rule written to anticipate one is a rule to
+ maintain. The line drawn instead:
+
+ * a declaration whose mistake would **delete the wrong rows, or silently
+ skip cleanup**, is refused, because neither announces itself;
+ * a declaration that merely **fails loudly** is left to fail. A predicate
+ the cleanup cannot build raises per row and is counted as a cascade
+ failure; an ordering a foreign key refuses surfaces as an integrity
+ error, which the cascade reports as a blocked purge carrying the
+ ``cascade_integrity_failure`` reason code. Either way the attempt is
+ recorded, isolated to its own root by the scheduled task, and the
+ entity is retried on the next run.
+
+ The frame's own requirements are checked regardless, since the scan and
+ the locked claim read them before any policy code runs.
+ """
+ metadata: sa.MetaData = sa.inspect(policy.model).local_table.metadata
+ for dependency in policy.dependencies:
+ _validate_dependency_declaration(policy, dependency, metadata)
+ _validate_scanner_requirements(policy)
+ if not policy.walks_own_subtree:
+ # Both shapes break the same way under cleanup that reaches one level
+ # and resolves each predicate through the ownership path. Only a
+ # policy that declares it walks the sub-tree itself is exempt.
+ _validate_owned_traversal(policy)
+ _validate_recursive_ownership(policy)
def get_purge_policy(model: type[Any]) -> PurgeEntityPolicy:
"""Resolve a complete policy or reject an unsupported purge model."""
- return _validated_purge_policy(cast(Hashable, model))
+ resolved: _ResolvedRegistry = _resolved_registry()
+ try:
+ policy: PurgeEntityPolicy = resolved.registry[model]
+ except KeyError as ex:
+ raise ValueError(f"Unsupported purge model: {model.__name__}") from ex
+ if model not in resolved.validated:
Review Comment:
`resolved.validated` is a plain set mutated without a lock from concurrent threads, and built-in validation errors are re-raised on every call because a failed model is never marked validated. That is harmless but it re-runs discovery for each row.
##########
superset/commands/deletion_retention/purge_policy.py:
##########
@@ -860,16 +923,444 @@ def declare(
cleanup_permission=cleanup_dataset_permission,
),
}
- return validate_unique_root_policies(registry.values())
+ return tuple(registry.values())
+
+def _policy_label(policy: Any) -> str:
+ """A log-safe name for a payload whose shape is not yet established."""
+ model: Any = getattr(policy, "model", None)
+ return str(getattr(model, "__name__", type(model).__name__))
+
+
+def _host_policy_payload(provided: Any) -> tuple[Any, ...]:
+ """Materialize the provider's payload, or reject its shape.
+
+ Any iterable is accepted -- a list, a tuple, ``dict.values()``, a
+ generator -- since the documented contract is a sequence of policies, not
+ one particular container. Strings and bytes are excluded because they
+ iterate into characters, which would read as a sequence of bad members
+ rather than the wrong type. Materialized once: a generator cannot be
+ walked twice.
+
+ An exception raised *while reading* a lazy payload is left to propagate.
+ Walking a generator runs host code just as calling the provider does, so
+ the two are the same kind of failure and earn the same retry; only the
+ shape of a payload that arrived intact is settled for good.
+ """
+ if isinstance(provided, (str, bytes)) or not isinstance(provided, Iterable):
+ logger.error(
+ "purge_policy: %s returned %s; expected an iterable of PurgeEntityPolicy",
+ HOST_POLICIES_CONFIG_KEY,
+ type(provided).__name__,
+ )
+ return ()
+ return tuple(provided)
-@lru_cache(maxsize=None)
-def _validated_purge_policy(model: type[Any]) -> PurgeEntityPolicy:
- """Validate and return one root policy without blocking unrelated roots."""
+
+def _unqualified_table_conflict(policy: PurgeEntityPolicy) -> str | None:
+ """Return the first table a bare-name lookup would resolve wrongly.
+
+ Discovery records table identities by bare name while a ``MetaData`` keys
+ them by schema, so a declared ``child`` resolves to the default-schema
+ table even where the policy meant ``host.child`` -- and the cleanup would
+ delete rows belonging to unrelated roots. Carrying schema-qualified
+ identities through discovery and execution is the better fix; until then
+ the ambiguous shape is refused rather than silently mis-resolved.
+ """
+ root_table: sa.Table = sa.inspect(policy.model).local_table
+ metadata: sa.MetaData = root_table.metadata
+ keys_by_name: dict[str, list[str]] = {}
+ for table in metadata.tables.values():
+ keys_by_name.setdefault(table.name, []).append(table.key)
+ declared: set[str] = {root_table.name} | {
+ dependency.key.related_table
+ for dependency in policy.dependencies
+ # Version shadows are resolved by bare name as well, in
+ # _entity_version_targets.
+ if dependency.classification in _EXECUTABLE_CLASSIFICATIONS
+ or dependency.classification is DependencyClassification.VERSION_OWNED
+ }
+ for name in sorted(declared):
+ keys: list[str] = keys_by_name.get(name, [])
+ if len(keys) != 1 or keys[0] != name:
+ return name
+ return None
+
+
+def _admitted_host_policy(
+ candidate: Any,
+ builtin_roots: frozenset[type[Any]],
+ builtin_tables: frozenset[str],
+ builtin_types: frozenset[str],
+) -> PurgeEntityPolicy | None:
+ """Return *candidate* if it is a usable host policy, else ``None``.
+
+ Every check runs behind one handler rather than guarding each attribute:
+ the payload is host-supplied, so reading ``model`` or hashing it may
+ itself raise -- and an exception escaping here would break
+ ``purge_policy_registry`` for the built-in roots too, which is the
+ opposite of what this boundary is for.
+ """
try:
- policy: PurgeEntityPolicy = purge_policy_registry()[model]
- except KeyError as ex:
- raise ValueError(f"Unsupported purge model: {model.__name__}") from ex
+ if not isinstance(candidate, PurgeEntityPolicy):
+ logger.error(
+ "purge_policy: %s returned a %s; expected PurgeEntityPolicy",
+ HOST_POLICIES_CONFIG_KEY,
+ type(candidate).__name__,
+ )
+ return None
+ if not isinstance(candidate.model, type):
+ # Everything downstream treats the root as a mapped class:
+ # sa.inspect, the registry index, the duplicate count.
+ logger.error(
+ "purge_policy: host policy root %s is not a class",
+ _policy_label(candidate),
+ )
+ return None
+ if candidate.model in builtin_roots:
+ logger.error(
+ "purge_policy: host policy for built-in root %s ignored",
+ candidate.model.__name__,
+ )
+ return None
+ root_table: sa.Table = sa.inspect(candidate.model).local_table
+ if root_table.key in builtin_tables:
+ # A different class mapped onto a built-in table -- a subclass of
+ # Slice, say -- is not the built-in root by identity, but purging
+ # it runs host cleanup against this package's own rows. It also
+ # joins the soft-delete registry through the mixin, so the
+ # scheduled task would scan it.
+ logger.error(
+ "purge_policy: host policy root %s is mapped onto built-in "
+ "table %r; ignored",
+ candidate.model.__name__,
+ root_table.key,
+ )
+ return None
+ if not isinstance(candidate.entity_type, str) or not candidate.entity_type:
+ # Read as a string by the cascade, the audit record and every log
+ # line that names the root.
+ logger.error(
+ "purge_policy: host policy for %s declares entity_type %r",
+ candidate.model.__name__,
+ candidate.entity_type,
+ )
+ return None
+ if candidate.entity_type in builtin_types:
+ # Core branches on entity_type -- the dataset impact path, the tag
+ # object type, the dataset permission name -- so a host reusing
+ # one of its names would have that logic pointed at its own rows.
+ logger.error(
+ "purge_policy: host policy for %s claims the reserved entity type %r",
+ candidate.model.__name__,
+ candidate.entity_type,
+ )
+ return None
+ # Validated first, so a table the metadata simply does not contain is
+ # reported as such rather than as an ambiguous name.
+ unusable: set[str] = {
+ dependency.classification.value
+ for dependency in candidate.dependencies
+ if dependency.classification
+ in {
+ DependencyClassification.BLOCK,
+ DependencyClassification.LISTENER_EFFECT,
+ }
+ }
+ if unusable:
+ # Two different reasons, both about identities this package owns.
+ # A stock listener action decides what to delete from a built-in
+ # entity type, so declared by a host it is accepted and then never
+ # runs. A blocker *would* be applied -- validate_deletion_allowed
+ # is generic over the declarations -- but its reason code lands in
+ # the purge audit record, whose vocabulary is a closed set pinned
+ # by test. A host refuses a purge from its own validator instead,
+ # where the code it records is plainly its own.
+ raise RuntimeError(
+ f"{', '.join(sorted(unusable))} dependencies are not available "
+ "to a host root"
+ )
+ _validated_policy(candidate)
+ conflict: str | None = _unqualified_table_conflict(candidate)
+ if conflict is not None:
+ raise RuntimeError(
+ f"table {conflict!r} cannot be identified unambiguously by "
+ "name; schema-qualified roots are not supported"
+ )
+ except Exception as ex: # pylint: disable=broad-except
+ # Validated at admission rather than when a row is purged: a
+ # declaration the cleanup cannot execute would otherwise fail once per
+ # eligible row, counted as a cascade failure, and stay invisible until
+ # something aged past the window.
+ logger.error(
+ "purge_policy: host policy for %s rejected: %s",
+ _policy_label(candidate),
+ ex,
+ )
+ return None
+ return candidate
+
+
+def _host_purge_policies(
+ provider: Callable[[], Any] | None,
+) -> tuple[PurgeEntityPolicy, ...] | None:
+ """Return the purge policies a host installed for its own roots.
+
+ A host distribution can carry ``SoftDeleteMixin`` entities this package
+ cannot import. The retention task discovers those entities through the
+ mixin registry, so without a policy they reach the cascade as an
+ unsupported model. The host therefore declares their purge behavior and
+ installs it under ``PURGE_POLICIES_FUNC``.
+
+ Host boundary: everything about a host policy is settled here, where it is
+ admitted -- an unavailable provider, a payload that is not an iterable of
+ policies, a root that is not a class, a collision with a root or entity
+ type declared in this package, two declarations for one root, and a
+ declaration the shared cleanup could not execute. Each is logged and
+ dropped, so that root is reported as unsupported instead of failing row by
+ row in a scheduled run. A broken host declaration must not take the purge
+ down with it, and must never redefine how a chart, dashboard or dataset is
+ purged.
+
+ Returns ``None`` for the failures that may be transient -- the provider
+ could not be called, or a lazy payload raised while being read. The caller
+ then publishes the built-in roots with a short deadline, so the next
+ resolution after it tries again rather than treating one bad moment as the
+ answer for the life of the process. A payload that arrived intact but is
+ the wrong shape is a host bug that will not fix itself, so it is published
+ as an empty result instead.
+ """
+ if provider is None:
+ return ()
+ if not callable(provider):
+ # Static misconfiguration, not an outage: reported once rather than
+ # retried with a traceback for the life of the process.
+ logger.error(
+ "purge_policy: %s is %s, not a callable returning policies",
+ HOST_POLICIES_CONFIG_KEY,
+ type(provider).__name__,
+ )
+ return ()
+ try:
+ # Both the call and reading what it returns run host code, so a lazy
+ # payload that raises mid-walk is the same kind of failure as a
+ # provider that raises outright, and earns the same retry.
+ candidates: tuple[Any, ...] = _host_policy_payload(provider())
+ except Exception: # pylint: disable=broad-except
+ logger.exception(
+ "purge_policy: %s is unavailable; keeping built-in roots only",
+ HOST_POLICIES_CONFIG_KEY,
+ )
+ return None
+
+ builtin_policies: tuple[PurgeEntityPolicy, ...] = _builtin_purge_policies()
+ builtin_roots: frozenset[type[Any]] = frozenset(
+ policy.model for policy in builtin_policies
+ )
+ builtin_tables: frozenset[str] = frozenset(
+ sa.inspect(policy.model).local_table.key for policy in builtin_policies
+ )
+ builtin_types: frozenset[str] = frozenset(
+ policy.entity_type for policy in builtin_policies
+ )
+ admitted: list[PurgeEntityPolicy] = [
+ policy
+ for candidate in candidates
+ if (
+ policy := _admitted_host_policy(
+ candidate, builtin_roots, builtin_tables, builtin_types
+ )
+ )
+ is not None
+ ]
+
+ # Counted after admission, where every root is known to be a class and so
+ # hashable. Which of two declarations for one root is authoritative is
+ # undecidable, and the loser would still delete rows, so both are dropped:
+ # the root is reported as unsupported, which is the recoverable outcome.
+ # It also keeps the duplicate away from validate_unique_root_policies,
+ # whose ValueError would abort the whole scheduled run.
+ counts: dict[type[Any], int] = {}
+ for policy in admitted:
+ counts[policy.model] = counts.get(policy.model, 0) + 1
+ for model, count in counts.items():
+ if count > 1:
+ logger.error(
+ "purge_policy: %s declares %d policies for %s; ignoring all of them",
+ HOST_POLICIES_CONFIG_KEY,
+ count,
+ model.__name__,
+ )
+ return tuple(policy for policy in admitted if counts[policy.model] == 1)
+
+
+#: Sentinel distinguishing "no index resolved yet" from an index resolved for
+#: no provider, which is the ordinary case.
+_UNRESOLVED: Any = object()
+
+
+@dataclass
+class _ResolvedRegistry:
+ """The index built for one provider, with the roots validated so far.
+
+ Matched to its provider by *identity* rather than keyed in an
+ ``lru_cache``: a cache key would have to be hashed, and a host callable
+ implemented as a mutable dataclass instance is unhashable. That raises
+ before the host boundary can isolate the host's mistake, which would
+ stop the built-in roots purging too.
+ """
+
+ provider: Any = _UNRESOLVED
+ registry: Mapping[type[Any], PurgeEntityPolicy] = field(
+ default_factory=lambda: MappingProxyType({})
+ )
+ validated: set[type[Any]] = field(default_factory=set)
+ #: Set only on a snapshot published because the provider could not be
+ #: called. Until this deadline the snapshot answers reads; after it, the
+ #: provider is tried again.
+ retry_after: float | None = None
+
+
+#: Key the resolved index lives under on the Flask app. Per app rather than
+#: per process: two apps in one process have different configs, and a single
+#: global snapshot would thrash between them -- re-resolving, and re-invoking
+#: the provider, on every read.
+_REGISTRY_EXTENSION_KEY: str = "deletion_retention_purge_registry"
+
+#: How long a snapshot published after a provider failure answers reads
+#: before the provider is tried again. Bounds both extremes: a failure is
+#: neither remembered for the life of the process nor retried on every read --
+#: at roughly three reads per purged entity, retrying per read meant tens of
+#: thousands of provider calls and tracebacks in a single pass.
+_PROVIDER_RETRY_SECONDS: float = 60.0
+
+#: Per-thread marker for "this thread is already resolving". A host provider
+#: may build its policy from a built-in one -- ``replace(get_purge_policy(
+#: Slice), model=HostRoot, ...)`` is the obvious way to borrow the stock
+#: callbacks -- which re-enters resolution while the first call is still in
+#: the provider. The nested call is handed the state already published
+#: instead, which always carries the built-in roots.
+_RESOLVING: threading.local = threading.local()
+
+
+def _resolved_registry() -> _ResolvedRegistry:
+ """Return the state resolved for the installed provider, building it once.
+
+ Readers take a snapshot and a rebuild publishes a new one, so a caller
+ always sees an index and its validation record together. Publishing is a
+ single rebind, which is what makes it safe to do this without a lock: two
+ concurrent first uses may each invoke the provider and each publish, but
+ the two snapshots are equivalent and neither is ever seen half-built.
+ Paying for a duplicate provider call is the deliberate trade -- a lock
+ here deadlocks the moment a provider resolves a built-in policy of its
+ own, and a hung purge is worse than a repeated call.
+ """
+ if not has_app_context():
+ return _builtin_registry()
+ provider: Callable[[], Any] | None = current_app.config.get(
+ HOST_POLICIES_CONFIG_KEY
+ )
+ published: _ResolvedRegistry | None = current_app.extensions.get(
+ _REGISTRY_EXTENSION_KEY
+ )
+ if published is None:
+ published = _builtin_registry()
+ current_app.extensions[_REGISTRY_EXTENSION_KEY] = published
+ if published.provider is provider and (
Review Comment:
The published snapshot is only reused when `published.provider is provider`, so a provider that is a fresh object on each `config.get` (a bound method, or a `functools.partial` built inside a property) re-resolves and re-invokes host code on every read.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(retention): let a host install purge policies for its own soft-delete roots [superset]
Posted by "bito-code-review[bot] (via GitHub)" <gi...@apache.org>.
bito-code-review[bot] commented on PR #44892:
URL: https://github.com/apache/superset/pull/44892#issuecomment-5955118531
AI Code Review is in progress (usually takes 3 to 15 minutes unless it's a very large PR).
<details>
<summary>
Bito Usage Guide
</summary>
**Commands**
Type the following command in the pull request comment and save the comment.
- `/review` - Manually triggers an incremental AI Review.
- `/review full` - Manually triggers a full AI Review.
- `/pause` - Pauses automatic reviews on this pull request.
- `/resume` - Resumes automatic reviews.
- `/resolve` - Marks all Bito-posted review comments as resolved.
- `/abort` - Cancels all in-progress reviews.
Refer to the <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/available-commands" target="_blank" rel="noopener noreferrer">documentation</a> for additional commands.
**Configuration**
This repository uses `Superset` You can customize the agent settings <a href="https://alpha.bito.ai/home/ai-agents/code-review-agent" target="_blank" rel="noopener noreferrer">here</a> or contact your Bito workspace admin at evan@preset.io.
**Documentation & Help**
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/getting-started/install-run-using-bito-cloud/create-or-customize-an-agent-instance" target="_blank" rel="noopener noreferrer">Customize agent settings</a>
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/implementing-custom-code-review-rules" target="_blank" rel="noopener noreferrer">Review rules</a>
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent" target="_blank" rel="noopener noreferrer">General documentation</a>
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/faqs" target="_blank" rel="noopener noreferrer">FAQ</a>
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(retention): let a host install purge policies for its own soft-delete roots [superset]
Posted by "bito-code-review[bot] (via GitHub)" <gi...@apache.org>.
bito-code-review[bot] commented on PR #44892:
URL: https://github.com/apache/superset/pull/44892#issuecomment-5996206134
AI Code Review is in progress (usually takes 3 to 15 minutes unless it's a very large PR).
<details>
<summary>
Bito Usage Guide
</summary>
**Commands**
Type the following command in the pull request comment and save the comment.
- `/review` - Manually triggers an incremental AI Review.
- `/review full` - Manually triggers a full AI Review.
- `/pause` - Pauses automatic reviews on this pull request.
- `/resume` - Resumes automatic reviews.
- `/resolve` - Marks all Bito-posted review comments as resolved.
- `/abort` - Cancels all in-progress reviews.
Refer to the <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/available-commands" target="_blank" rel="noopener noreferrer">documentation</a> for additional commands.
**Configuration**
This repository uses `Superset` You can customize the agent settings <a href="https://alpha.bito.ai/home/ai-agents/code-review-agent" target="_blank" rel="noopener noreferrer">here</a> or contact your Bito workspace admin at evan@preset.io.
**Documentation & Help**
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/getting-started/install-run-using-bito-cloud/create-or-customize-an-agent-instance" target="_blank" rel="noopener noreferrer">Customize agent settings</a>
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/implementing-custom-code-review-rules" target="_blank" rel="noopener noreferrer">Review rules</a>
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent" target="_blank" rel="noopener noreferrer">General documentation</a>
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/faqs" target="_blank" rel="noopener noreferrer">FAQ</a>
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(retention): let a host install purge policies for its own soft-delete roots [superset]
Posted by "msyavuz (via GitHub)" <gi...@apache.org>.
msyavuz commented on code in PR #44892:
URL: https://github.com/apache/superset/pull/44892#discussion_r4195219815
##########
superset/commands/deletion_retention/purge_policy.py:
##########
@@ -860,16 +913,349 @@ def declare(
cleanup_permission=cleanup_dataset_permission,
),
}
- return validate_unique_root_policies(registry.values())
+ return tuple(registry.values())
+
+
+def _policy_label(policy: Any) -> str:
+ """A log-safe name for a payload whose shape is not yet established."""
+ model: Any = getattr(policy, "model", None)
+ return str(getattr(model, "__name__", type(model).__name__))
+
+def _host_policy_payload(provided: Any) -> tuple[Any, ...] | None:
+ """Materialize the provider's payload, or reject it.
-@lru_cache(maxsize=None)
-def _validated_purge_policy(model: type[Any]) -> PurgeEntityPolicy:
- """Validate and return one root policy without blocking unrelated roots."""
+ Any iterable is accepted -- a list, a tuple, ``dict.values()``, a
+ generator -- since the documented contract is a sequence of policies, not
+ one particular container. Strings and bytes are excluded because they
+ iterate into characters, which would read as a sequence of bad members
+ rather than the wrong type. Materialized once: a generator cannot be
+ walked twice.
+ """
+ if isinstance(provided, (str, bytes)) or not isinstance(provided, Iterable):
+ logger.error(
+ "purge_policy: %s returned %s; expected an iterable of PurgeEntityPolicy",
+ HOST_POLICIES_CONFIG_KEY,
+ type(provided).__name__,
+ )
+ return None
try:
- policy: PurgeEntityPolicy = purge_policy_registry()[model]
- except KeyError as ex:
- raise ValueError(f"Unsupported purge model: {model.__name__}") from ex
+ return tuple(provided)
+ except Exception: # pylint: disable=broad-except
+ logger.exception(
+ "purge_policy: %s could not be read; keeping built-in roots only",
+ HOST_POLICIES_CONFIG_KEY,
+ )
+ return None
+
+
+def _unqualified_table_conflict(policy: PurgeEntityPolicy) -> str | None:
+ """Return the first table a bare-name lookup would resolve wrongly.
+
+ Discovery records table identities by bare name while a ``MetaData`` keys
+ them by schema, so a declared ``child`` resolves to the default-schema
+ table even where the policy meant ``host.child`` -- and the cleanup would
+ delete rows belonging to unrelated roots. Carrying schema-qualified
+ identities through discovery and execution is the better fix; until then
+ the ambiguous shape is refused rather than silently mis-resolved.
+ """
+ root_table: sa.Table = sa.inspect(policy.model).local_table
+ metadata: sa.MetaData = root_table.metadata
+ keys_by_name: dict[str, list[str]] = {}
+ for table in metadata.tables.values():
+ keys_by_name.setdefault(table.name, []).append(table.key)
+ declared: set[str] = {root_table.name} | {
+ dependency.key.related_table
+ for dependency in policy.dependencies
+ if dependency.classification in _EXECUTABLE_CLASSIFICATIONS
+ }
+ for name in sorted(declared):
+ keys: list[str] = keys_by_name.get(name, [])
+ if len(keys) != 1 or keys[0] != name:
+ return name
+ return None
+
+
+def _admitted_host_policy(
+ candidate: Any, builtin_roots: frozenset[type[Any]], builtin_types: frozenset[str]
+) -> PurgeEntityPolicy | None:
+ """Return *candidate* if it is a usable host policy, else ``None``.
+
+ Every check runs behind one handler rather than guarding each attribute:
+ the payload is host-supplied, so reading ``model`` or hashing it may
+ itself raise -- and an exception escaping here would break
+ ``purge_policy_registry`` for the built-in roots too, which is the
+ opposite of what this boundary is for.
+ """
+ try:
+ if not isinstance(candidate, PurgeEntityPolicy):
+ logger.error(
+ "purge_policy: %s returned a %s; expected PurgeEntityPolicy",
+ HOST_POLICIES_CONFIG_KEY,
+ type(candidate).__name__,
+ )
+ return None
+ if not isinstance(candidate.model, type):
+ # Everything downstream treats the root as a mapped class:
+ # sa.inspect, the registry index, the duplicate count.
+ logger.error(
+ "purge_policy: host policy root %s is not a class",
+ _policy_label(candidate),
+ )
+ return None
+ if candidate.model in builtin_roots:
+ logger.error(
+ "purge_policy: host policy for built-in root %s ignored",
+ candidate.model.__name__,
+ )
+ return None
+ if candidate.entity_type in builtin_types:
+ # Core branches on entity_type -- the dataset impact path, the tag
+ # object type, the dataset permission name -- so a host reusing
+ # one of its names would have that logic pointed at its own rows.
+ logger.error(
+ "purge_policy: host policy for %s claims the reserved entity type %r",
+ candidate.model.__name__,
+ candidate.entity_type,
+ )
+ return None
+ # Validated first, so a table the metadata simply does not contain is
+ # reported as such rather than as an ambiguous name.
+ unusable: set[str] = {
+ dependency.classification.value
+ for dependency in candidate.dependencies
+ if dependency.classification
+ in {
+ DependencyClassification.BLOCK,
+ DependencyClassification.LISTENER_EFFECT,
+ }
+ }
+ if unusable:
+ # Both resolve from core's own identities: the stock listener
+ # actions decide what to delete from a built-in entity type, and
+ # blockers are applied by the stock validator. Declared by a host,
+ # they are accepted and then never carried out -- silence rather
+ # than a failure, which is the one outcome worth refusing. A host
+ # blocks by raising PurgeBlockedError from its own validate.
+ raise RuntimeError(
Review Comment:
**major**: Rejecting BLOCK and LISTENER_EFFECT stops a host from declaring them, but the built-in stock cleanup still runs on host roots. Tag, permission and other core listener cleanup keyed off entity_type won't run, so host rows can be purged with core-side references left dangling and no error.
##########
superset/commands/deletion_retention/purge_policy.py:
##########
@@ -860,16 +913,349 @@ def declare(
cleanup_permission=cleanup_dataset_permission,
),
}
- return validate_unique_root_policies(registry.values())
+ return tuple(registry.values())
+
+
+def _policy_label(policy: Any) -> str:
+ """A log-safe name for a payload whose shape is not yet established."""
+ model: Any = getattr(policy, "model", None)
+ return str(getattr(model, "__name__", type(model).__name__))
+
+def _host_policy_payload(provided: Any) -> tuple[Any, ...] | None:
+ """Materialize the provider's payload, or reject it.
-@lru_cache(maxsize=None)
-def _validated_purge_policy(model: type[Any]) -> PurgeEntityPolicy:
- """Validate and return one root policy without blocking unrelated roots."""
+ Any iterable is accepted -- a list, a tuple, ``dict.values()``, a
+ generator -- since the documented contract is a sequence of policies, not
+ one particular container. Strings and bytes are excluded because they
+ iterate into characters, which would read as a sequence of bad members
+ rather than the wrong type. Materialized once: a generator cannot be
+ walked twice.
+ """
+ if isinstance(provided, (str, bytes)) or not isinstance(provided, Iterable):
+ logger.error(
+ "purge_policy: %s returned %s; expected an iterable of PurgeEntityPolicy",
+ HOST_POLICIES_CONFIG_KEY,
+ type(provided).__name__,
+ )
+ return None
try:
- policy: PurgeEntityPolicy = purge_policy_registry()[model]
- except KeyError as ex:
- raise ValueError(f"Unsupported purge model: {model.__name__}") from ex
+ return tuple(provided)
+ except Exception: # pylint: disable=broad-except
+ logger.exception(
+ "purge_policy: %s could not be read; keeping built-in roots only",
+ HOST_POLICIES_CONFIG_KEY,
+ )
+ return None
+
+
+def _unqualified_table_conflict(policy: PurgeEntityPolicy) -> str | None:
+ """Return the first table a bare-name lookup would resolve wrongly.
+
+ Discovery records table identities by bare name while a ``MetaData`` keys
+ them by schema, so a declared ``child`` resolves to the default-schema
+ table even where the policy meant ``host.child`` -- and the cleanup would
+ delete rows belonging to unrelated roots. Carrying schema-qualified
+ identities through discovery and execution is the better fix; until then
+ the ambiguous shape is refused rather than silently mis-resolved.
+ """
+ root_table: sa.Table = sa.inspect(policy.model).local_table
+ metadata: sa.MetaData = root_table.metadata
+ keys_by_name: dict[str, list[str]] = {}
+ for table in metadata.tables.values():
+ keys_by_name.setdefault(table.name, []).append(table.key)
+ declared: set[str] = {root_table.name} | {
+ dependency.key.related_table
+ for dependency in policy.dependencies
+ if dependency.classification in _EXECUTABLE_CLASSIFICATIONS
+ }
+ for name in sorted(declared):
+ keys: list[str] = keys_by_name.get(name, [])
+ if len(keys) != 1 or keys[0] != name:
+ return name
+ return None
+
+
+def _admitted_host_policy(
+ candidate: Any, builtin_roots: frozenset[type[Any]], builtin_types: frozenset[str]
+) -> PurgeEntityPolicy | None:
+ """Return *candidate* if it is a usable host policy, else ``None``.
+
+ Every check runs behind one handler rather than guarding each attribute:
+ the payload is host-supplied, so reading ``model`` or hashing it may
+ itself raise -- and an exception escaping here would break
+ ``purge_policy_registry`` for the built-in roots too, which is the
+ opposite of what this boundary is for.
+ """
+ try:
+ if not isinstance(candidate, PurgeEntityPolicy):
+ logger.error(
+ "purge_policy: %s returned a %s; expected PurgeEntityPolicy",
+ HOST_POLICIES_CONFIG_KEY,
+ type(candidate).__name__,
+ )
+ return None
+ if not isinstance(candidate.model, type):
+ # Everything downstream treats the root as a mapped class:
+ # sa.inspect, the registry index, the duplicate count.
+ logger.error(
+ "purge_policy: host policy root %s is not a class",
+ _policy_label(candidate),
+ )
+ return None
+ if candidate.model in builtin_roots:
+ logger.error(
+ "purge_policy: host policy for built-in root %s ignored",
+ candidate.model.__name__,
+ )
+ return None
+ if candidate.entity_type in builtin_types:
+ # Core branches on entity_type -- the dataset impact path, the tag
+ # object type, the dataset permission name -- so a host reusing
+ # one of its names would have that logic pointed at its own rows.
+ logger.error(
+ "purge_policy: host policy for %s claims the reserved entity type %r",
+ candidate.model.__name__,
+ candidate.entity_type,
+ )
+ return None
+ # Validated first, so a table the metadata simply does not contain is
+ # reported as such rather than as an ambiguous name.
+ unusable: set[str] = {
+ dependency.classification.value
+ for dependency in candidate.dependencies
+ if dependency.classification
+ in {
+ DependencyClassification.BLOCK,
+ DependencyClassification.LISTENER_EFFECT,
+ }
+ }
+ if unusable:
+ # Both resolve from core's own identities: the stock listener
+ # actions decide what to delete from a built-in entity type, and
+ # blockers are applied by the stock validator. Declared by a host,
+ # they are accepted and then never carried out -- silence rather
+ # than a failure, which is the one outcome worth refusing. A host
+ # blocks by raising PurgeBlockedError from its own validate.
+ raise RuntimeError(
+ f"{', '.join(sorted(unusable))} dependencies cannot be carried "
+ "out for a host root"
+ )
+ _validated_policy(candidate)
+ conflict: str | None = _unqualified_table_conflict(candidate)
+ if conflict is not None:
+ raise RuntimeError(
+ f"table {conflict!r} cannot be identified unambiguously by "
+ "name; schema-qualified roots are not supported"
+ )
+ except Exception as ex: # pylint: disable=broad-except
+ # Validated at admission rather than when a row is purged: a
+ # declaration the cleanup cannot execute would otherwise fail once per
+ # eligible row, counted as a cascade failure, and stay invisible until
+ # something aged past the window.
+ logger.error(
+ "purge_policy: host policy for %s rejected: %s",
+ _policy_label(candidate),
+ ex,
+ )
+ return None
+ return candidate
+
+
+def _host_purge_policies(
+ provider: Callable[[], Any] | None,
+) -> tuple[PurgeEntityPolicy, ...] | None:
+ """Return the purge policies a host installed for its own roots.
+
+ A host distribution can carry ``SoftDeleteMixin`` entities this package
+ cannot import. The retention task discovers those entities through the
+ mixin registry, so without a policy they reach the cascade as an
+ unsupported model. The host therefore declares their purge behavior and
+ installs it under ``PURGE_POLICIES_FUNC``.
+
+ Host boundary: everything about a host policy is settled here, where it is
+ admitted -- an unavailable provider, a payload that is not an iterable of
+ policies, a root that is not a class, a collision with a root or entity
+ type declared in this package, two declarations for one root, and a
+ declaration the shared cleanup could not execute. Each is logged and
+ dropped, so that root is reported as unsupported instead of failing row by
+ row in a scheduled run. A broken host declaration must not take the purge
+ down with it, and must never redefine how a chart, dashboard or dataset is
+ purged.
+
+ Returns ``None`` for the one failure that may be transient -- the provider
+ itself could not be called. The caller then serves the built-in roots
+ *without* publishing, so the next resolution tries again instead of
+ freezing a bad moment in for the life of the process. A payload that is
+ merely malformed is a host bug that will not fix itself, so it is
+ published as an empty result rather than retried on every read.
+ """
+ if provider is None:
+ return ()
+ try:
+ provided: Any = provider()
+ except Exception: # pylint: disable=broad-except
+ logger.exception(
+ "purge_policy: %s is unavailable; keeping built-in roots only",
+ HOST_POLICIES_CONFIG_KEY,
+ )
+ return None
+ candidates: tuple[Any, ...] | None = _host_policy_payload(provided)
+ if candidates is None:
+ return ()
+
+ builtin_policies: tuple[PurgeEntityPolicy, ...] = _builtin_purge_policies()
+ builtin_roots: frozenset[type[Any]] = frozenset(
+ policy.model for policy in builtin_policies
+ )
+ builtin_types: frozenset[str] = frozenset(
+ policy.entity_type for policy in builtin_policies
+ )
+ admitted: list[PurgeEntityPolicy] = [
+ policy
+ for candidate in candidates
+ if (policy := _admitted_host_policy(candidate, builtin_roots, builtin_types))
+ is not None
+ ]
+
+ # Counted after admission, where every root is known to be a class and so
+ # hashable. Which of two declarations for one root is authoritative is
+ # undecidable, and the loser would still delete rows, so both are dropped:
+ # the root is reported as unsupported, which is the recoverable outcome.
+ # It also keeps the duplicate away from validate_unique_root_policies,
+ # whose ValueError would abort the whole scheduled run.
+ counts: dict[type[Any], int] = {}
+ for policy in admitted:
+ counts[policy.model] = counts.get(policy.model, 0) + 1
+ for model, count in counts.items():
+ if count > 1:
+ logger.error(
+ "purge_policy: %s declares %d policies for %s; ignoring all of them",
+ HOST_POLICIES_CONFIG_KEY,
+ count,
+ model.__name__,
+ )
+ return tuple(policy for policy in admitted if counts[policy.model] == 1)
+
+
+#: Sentinel distinguishing "no index resolved yet" from an index resolved for
+#: no provider, which is the ordinary case.
+_UNRESOLVED: Any = object()
+
+
+@dataclass
+class _ResolvedRegistry:
+ """The index built for one provider, with the roots validated so far.
+
+ Compared by provider *identity* rather than memoized with ``lru_cache``: a
+ cache keyed on the provider would hash it, and a host callable
+ implemented as a mutable dataclass instance is unhashable. That would
+ raise here -- before the host boundary could isolate the host's mistake --
+ and stop the built-in roots purging too.
+ """
+
+ provider: Any = _UNRESOLVED
+ registry: Mapping[type[Any], PurgeEntityPolicy] = field(
+ default_factory=lambda: MappingProxyType({})
+ )
+ validated: set[type[Any]] = field(default_factory=set)
+
+
+_RESOLVED: _ResolvedRegistry = _ResolvedRegistry()
+
+#: Per-thread marker for "this thread is already resolving". A host provider
+#: may build its policy from a built-in one -- ``replace(get_purge_policy(
+#: Slice), model=HostRoot, ...)`` is the obvious way to borrow the stock
+#: callbacks -- which re-enters resolution while the first call is still in
+#: the provider. The nested call is handed the state already published
+#: instead, which always carries the built-in roots.
+_RESOLVING: threading.local = threading.local()
+
+
+def _resolved_registry() -> _ResolvedRegistry:
+ """Return the state resolved for the installed provider, building it once.
+
+ Readers take a snapshot and a rebuild publishes a new one, so a caller
+ always sees an index and its validation record together. Publishing is a
+ single rebind, which is what makes it safe to do this without a lock: two
+ concurrent first uses may each invoke the provider and each publish, but
+ the two snapshots are equivalent and neither is ever seen half-built.
+ Paying for a duplicate provider call is the deliberate trade -- a lock
+ here deadlocks the moment a provider resolves a built-in policy of its
+ own, and a hung purge is worse than a repeated call.
+ """
+ provider: Callable[[], Any] | None = (
+ current_app.config.get(HOST_POLICIES_CONFIG_KEY) if has_app_context() else None
+ )
+ resolved: _ResolvedRegistry = _RESOLVED
+ if resolved.provider is provider:
+ return resolved
+ if getattr(_RESOLVING, "active", False):
+ # A nested resolution: the provider is building its policy from a
+ # built-in one. It needs the roots this package declares -- not
+ # whatever happens to be published, which on a first resolution is
+ # still the empty initial snapshot.
+ return _builtin_registry()
Review Comment:
**major**: A nested resolution returns `_builtin_registry()`, which is `lru_cache`d and carries `provider=_UNRESOLVED`. A provider that calls `get_purge_policy` on a built-in root therefore validates against a separate `validated` set. Together with the unpublished transient-failure path, this means built-in roots are validated twice and the two snapshots can disagree.
##########
superset/tasks/deletion_retention.py:
##########
@@ -192,44 +244,61 @@ def _purge_impl(window_days: int, dry_run: bool) -> dict[str, Any]:
stats_logger_manager.instance.gauge(
f"{_METRIC_PREFIX}.blocked_by_reference", blocked
)
+ if scan_failures:
+ stats_logger_manager.instance.gauge(
+ f"{_METRIC_PREFIX}.scan_failures", scan_failures
+ )
stats = {
"purged": purged,
"cascade_failures": failures,
"blocked_by_reference": blocked,
"unsupported_models": unsupported_models,
+ "scan_failures": scan_failures,
}
logger.info("deletion_retention: %s", stats)
return stats
def _purge_model(
model: type[SoftDeleteMixin], cutoff: datetime, dry_run: bool
-) -> tuple[int, int, int, int]:
+) -> tuple[int, int, int, int, int]:
"""Process one model's eligible rows. Returns ``(purged, would_purge,
- failures, blocked)``. A single entity's blocked/failed cascade never aborts
- the batch."""
+ failures, blocked, scan_failures)``. A single entity's blocked/failed
+ cascade never aborts the batch.
+
+ A failure in the eligible-id scan itself -- a column it cannot read, a
+ transient database error between pages -- ends this model's pass and is
+ reported, but the counts earned before it are kept: by then those rows are
+ committed deletions, and a summary that omitted them would understate what
+ the run actually removed.
+ """
entity_type = _model_table_name(model)
- purged = would = failures = blocked = 0
- for id_batch in _iter_eligible_ids(model, cutoff, _BATCH):
- if dry_run:
- would += len(id_batch)
- continue
- for entity_id in id_batch:
- try:
- result = _purge_one(model, entity_id, cutoff)
- if result is not None and result.purged:
- purged += 1
- elif result is not None and result.blocked_reason is not None:
- blocked += 1
- except Exception: # pylint: disable=broad-except
- db.session.rollback() # pylint: disable=consider-using-transaction
- failures += 1
- logger.exception(
- "deletion_retention: cascade failed for %s id=%s",
- entity_type,
- entity_id,
- )
- return purged, would, failures, blocked
+ purged = would = failures = blocked = scan_failures = 0
+ try:
+ for id_batch in _iter_eligible_ids(model, cutoff, _BATCH):
+ if dry_run:
+ would += len(id_batch)
+ continue
+ for entity_id in id_batch:
+ try:
+ result = _purge_one(model, entity_id, cutoff)
+ if result is not None and result.purged:
+ purged += 1
+ elif result is not None and result.blocked_reason is not None:
+ blocked += 1
+ except Exception: # pylint: disable=broad-except
+ db.session.rollback() # pylint: disable=consider-using-transaction
+ failures += 1
+ logger.exception(
+ "deletion_retention: cascade failed for %s id=%s",
+ entity_type,
+ entity_id,
+ )
+ except Exception: # pylint: disable=broad-except
Review Comment:
**minor**: The scan-failure handler rolls back the session after earlier pages have committed, then moves on. A failure repeated every night for one root only shows up as a gauge, and the run still returns normally. Nothing in the task's return status or exit code flags it.
##########
superset/commands/deletion_retention/purge_policy.py:
##########
@@ -860,16 +913,349 @@ def declare(
cleanup_permission=cleanup_dataset_permission,
),
}
- return validate_unique_root_policies(registry.values())
+ return tuple(registry.values())
+
+
+def _policy_label(policy: Any) -> str:
+ """A log-safe name for a payload whose shape is not yet established."""
+ model: Any = getattr(policy, "model", None)
+ return str(getattr(model, "__name__", type(model).__name__))
+
+def _host_policy_payload(provided: Any) -> tuple[Any, ...] | None:
+ """Materialize the provider's payload, or reject it.
-@lru_cache(maxsize=None)
-def _validated_purge_policy(model: type[Any]) -> PurgeEntityPolicy:
- """Validate and return one root policy without blocking unrelated roots."""
+ Any iterable is accepted -- a list, a tuple, ``dict.values()``, a
+ generator -- since the documented contract is a sequence of policies, not
+ one particular container. Strings and bytes are excluded because they
+ iterate into characters, which would read as a sequence of bad members
+ rather than the wrong type. Materialized once: a generator cannot be
+ walked twice.
+ """
+ if isinstance(provided, (str, bytes)) or not isinstance(provided, Iterable):
+ logger.error(
+ "purge_policy: %s returned %s; expected an iterable of PurgeEntityPolicy",
+ HOST_POLICIES_CONFIG_KEY,
+ type(provided).__name__,
+ )
+ return None
try:
- policy: PurgeEntityPolicy = purge_policy_registry()[model]
- except KeyError as ex:
- raise ValueError(f"Unsupported purge model: {model.__name__}") from ex
+ return tuple(provided)
+ except Exception: # pylint: disable=broad-except
+ logger.exception(
+ "purge_policy: %s could not be read; keeping built-in roots only",
+ HOST_POLICIES_CONFIG_KEY,
+ )
+ return None
+
+
+def _unqualified_table_conflict(policy: PurgeEntityPolicy) -> str | None:
+ """Return the first table a bare-name lookup would resolve wrongly.
+
+ Discovery records table identities by bare name while a ``MetaData`` keys
+ them by schema, so a declared ``child`` resolves to the default-schema
+ table even where the policy meant ``host.child`` -- and the cleanup would
+ delete rows belonging to unrelated roots. Carrying schema-qualified
+ identities through discovery and execution is the better fix; until then
+ the ambiguous shape is refused rather than silently mis-resolved.
+ """
+ root_table: sa.Table = sa.inspect(policy.model).local_table
+ metadata: sa.MetaData = root_table.metadata
+ keys_by_name: dict[str, list[str]] = {}
+ for table in metadata.tables.values():
+ keys_by_name.setdefault(table.name, []).append(table.key)
+ declared: set[str] = {root_table.name} | {
+ dependency.key.related_table
+ for dependency in policy.dependencies
+ if dependency.classification in _EXECUTABLE_CLASSIFICATIONS
+ }
+ for name in sorted(declared):
+ keys: list[str] = keys_by_name.get(name, [])
+ if len(keys) != 1 or keys[0] != name:
+ return name
+ return None
+
+
+def _admitted_host_policy(
+ candidate: Any, builtin_roots: frozenset[type[Any]], builtin_types: frozenset[str]
+) -> PurgeEntityPolicy | None:
+ """Return *candidate* if it is a usable host policy, else ``None``.
+
+ Every check runs behind one handler rather than guarding each attribute:
+ the payload is host-supplied, so reading ``model`` or hashing it may
+ itself raise -- and an exception escaping here would break
+ ``purge_policy_registry`` for the built-in roots too, which is the
+ opposite of what this boundary is for.
+ """
+ try:
+ if not isinstance(candidate, PurgeEntityPolicy):
+ logger.error(
+ "purge_policy: %s returned a %s; expected PurgeEntityPolicy",
+ HOST_POLICIES_CONFIG_KEY,
+ type(candidate).__name__,
+ )
+ return None
+ if not isinstance(candidate.model, type):
+ # Everything downstream treats the root as a mapped class:
+ # sa.inspect, the registry index, the duplicate count.
+ logger.error(
+ "purge_policy: host policy root %s is not a class",
+ _policy_label(candidate),
+ )
+ return None
+ if candidate.model in builtin_roots:
+ logger.error(
+ "purge_policy: host policy for built-in root %s ignored",
+ candidate.model.__name__,
+ )
+ return None
+ if candidate.entity_type in builtin_types:
+ # Core branches on entity_type -- the dataset impact path, the tag
+ # object type, the dataset permission name -- so a host reusing
+ # one of its names would have that logic pointed at its own rows.
+ logger.error(
+ "purge_policy: host policy for %s claims the reserved entity type %r",
+ candidate.model.__name__,
+ candidate.entity_type,
+ )
+ return None
+ # Validated first, so a table the metadata simply does not contain is
+ # reported as such rather than as an ambiguous name.
+ unusable: set[str] = {
+ dependency.classification.value
+ for dependency in candidate.dependencies
+ if dependency.classification
+ in {
+ DependencyClassification.BLOCK,
+ DependencyClassification.LISTENER_EFFECT,
+ }
+ }
+ if unusable:
+ # Both resolve from core's own identities: the stock listener
+ # actions decide what to delete from a built-in entity type, and
+ # blockers are applied by the stock validator. Declared by a host,
+ # they are accepted and then never carried out -- silence rather
+ # than a failure, which is the one outcome worth refusing. A host
+ # blocks by raising PurgeBlockedError from its own validate.
+ raise RuntimeError(
+ f"{', '.join(sorted(unusable))} dependencies cannot be carried "
+ "out for a host root"
+ )
+ _validated_policy(candidate)
+ conflict: str | None = _unqualified_table_conflict(candidate)
+ if conflict is not None:
+ raise RuntimeError(
+ f"table {conflict!r} cannot be identified unambiguously by "
+ "name; schema-qualified roots are not supported"
+ )
+ except Exception as ex: # pylint: disable=broad-except
+ # Validated at admission rather than when a row is purged: a
+ # declaration the cleanup cannot execute would otherwise fail once per
+ # eligible row, counted as a cascade failure, and stay invisible until
+ # something aged past the window.
+ logger.error(
+ "purge_policy: host policy for %s rejected: %s",
+ _policy_label(candidate),
+ ex,
+ )
+ return None
+ return candidate
+
+
+def _host_purge_policies(
+ provider: Callable[[], Any] | None,
+) -> tuple[PurgeEntityPolicy, ...] | None:
+ """Return the purge policies a host installed for its own roots.
+
+ A host distribution can carry ``SoftDeleteMixin`` entities this package
+ cannot import. The retention task discovers those entities through the
+ mixin registry, so without a policy they reach the cascade as an
+ unsupported model. The host therefore declares their purge behavior and
+ installs it under ``PURGE_POLICIES_FUNC``.
+
+ Host boundary: everything about a host policy is settled here, where it is
+ admitted -- an unavailable provider, a payload that is not an iterable of
+ policies, a root that is not a class, a collision with a root or entity
+ type declared in this package, two declarations for one root, and a
+ declaration the shared cleanup could not execute. Each is logged and
+ dropped, so that root is reported as unsupported instead of failing row by
+ row in a scheduled run. A broken host declaration must not take the purge
+ down with it, and must never redefine how a chart, dashboard or dataset is
+ purged.
+
+ Returns ``None`` for the one failure that may be transient -- the provider
+ itself could not be called. The caller then serves the built-in roots
+ *without* publishing, so the next resolution tries again instead of
+ freezing a bad moment in for the life of the process. A payload that is
+ merely malformed is a host bug that will not fix itself, so it is
+ published as an empty result rather than retried on every read.
+ """
+ if provider is None:
+ return ()
+ try:
+ provided: Any = provider()
+ except Exception: # pylint: disable=broad-except
+ logger.exception(
+ "purge_policy: %s is unavailable; keeping built-in roots only",
+ HOST_POLICIES_CONFIG_KEY,
+ )
+ return None
+ candidates: tuple[Any, ...] | None = _host_policy_payload(provided)
+ if candidates is None:
+ return ()
+
+ builtin_policies: tuple[PurgeEntityPolicy, ...] = _builtin_purge_policies()
+ builtin_roots: frozenset[type[Any]] = frozenset(
+ policy.model for policy in builtin_policies
+ )
+ builtin_types: frozenset[str] = frozenset(
+ policy.entity_type for policy in builtin_policies
+ )
+ admitted: list[PurgeEntityPolicy] = [
+ policy
+ for candidate in candidates
+ if (policy := _admitted_host_policy(candidate, builtin_roots, builtin_types))
+ is not None
+ ]
+
+ # Counted after admission, where every root is known to be a class and so
+ # hashable. Which of two declarations for one root is authoritative is
+ # undecidable, and the loser would still delete rows, so both are dropped:
+ # the root is reported as unsupported, which is the recoverable outcome.
+ # It also keeps the duplicate away from validate_unique_root_policies,
+ # whose ValueError would abort the whole scheduled run.
+ counts: dict[type[Any], int] = {}
+ for policy in admitted:
+ counts[policy.model] = counts.get(policy.model, 0) + 1
+ for model, count in counts.items():
+ if count > 1:
+ logger.error(
+ "purge_policy: %s declares %d policies for %s; ignoring all of them",
+ HOST_POLICIES_CONFIG_KEY,
+ count,
+ model.__name__,
+ )
+ return tuple(policy for policy in admitted if counts[policy.model] == 1)
Review Comment:
**minor**: Host-supplied `delete_*` callbacks are executed inside the locked purge frame and are not checked at admission. A host that cloned the chart policy via `replace(...)` without overriding them would run chart-specific cleanup against its own table. Only the reserved `entity_type` is guarded.
##########
superset/commands/deletion_retention/purge_policy.py:
##########
@@ -860,16 +913,349 @@ def declare(
cleanup_permission=cleanup_dataset_permission,
),
}
- return validate_unique_root_policies(registry.values())
+ return tuple(registry.values())
+
+
+def _policy_label(policy: Any) -> str:
+ """A log-safe name for a payload whose shape is not yet established."""
+ model: Any = getattr(policy, "model", None)
+ return str(getattr(model, "__name__", type(model).__name__))
+
+def _host_policy_payload(provided: Any) -> tuple[Any, ...] | None:
+ """Materialize the provider's payload, or reject it.
-@lru_cache(maxsize=None)
-def _validated_purge_policy(model: type[Any]) -> PurgeEntityPolicy:
- """Validate and return one root policy without blocking unrelated roots."""
+ Any iterable is accepted -- a list, a tuple, ``dict.values()``, a
+ generator -- since the documented contract is a sequence of policies, not
+ one particular container. Strings and bytes are excluded because they
+ iterate into characters, which would read as a sequence of bad members
+ rather than the wrong type. Materialized once: a generator cannot be
+ walked twice.
+ """
+ if isinstance(provided, (str, bytes)) or not isinstance(provided, Iterable):
+ logger.error(
+ "purge_policy: %s returned %s; expected an iterable of PurgeEntityPolicy",
+ HOST_POLICIES_CONFIG_KEY,
+ type(provided).__name__,
+ )
+ return None
try:
- policy: PurgeEntityPolicy = purge_policy_registry()[model]
- except KeyError as ex:
- raise ValueError(f"Unsupported purge model: {model.__name__}") from ex
+ return tuple(provided)
+ except Exception: # pylint: disable=broad-except
+ logger.exception(
+ "purge_policy: %s could not be read; keeping built-in roots only",
+ HOST_POLICIES_CONFIG_KEY,
+ )
+ return None
+
+
+def _unqualified_table_conflict(policy: PurgeEntityPolicy) -> str | None:
+ """Return the first table a bare-name lookup would resolve wrongly.
+
+ Discovery records table identities by bare name while a ``MetaData`` keys
+ them by schema, so a declared ``child`` resolves to the default-schema
+ table even where the policy meant ``host.child`` -- and the cleanup would
+ delete rows belonging to unrelated roots. Carrying schema-qualified
+ identities through discovery and execution is the better fix; until then
+ the ambiguous shape is refused rather than silently mis-resolved.
+ """
+ root_table: sa.Table = sa.inspect(policy.model).local_table
+ metadata: sa.MetaData = root_table.metadata
+ keys_by_name: dict[str, list[str]] = {}
+ for table in metadata.tables.values():
+ keys_by_name.setdefault(table.name, []).append(table.key)
+ declared: set[str] = {root_table.name} | {
+ dependency.key.related_table
+ for dependency in policy.dependencies
+ if dependency.classification in _EXECUTABLE_CLASSIFICATIONS
+ }
+ for name in sorted(declared):
+ keys: list[str] = keys_by_name.get(name, [])
+ if len(keys) != 1 or keys[0] != name:
+ return name
+ return None
+
+
+def _admitted_host_policy(
+ candidate: Any, builtin_roots: frozenset[type[Any]], builtin_types: frozenset[str]
+) -> PurgeEntityPolicy | None:
+ """Return *candidate* if it is a usable host policy, else ``None``.
+
+ Every check runs behind one handler rather than guarding each attribute:
+ the payload is host-supplied, so reading ``model`` or hashing it may
+ itself raise -- and an exception escaping here would break
+ ``purge_policy_registry`` for the built-in roots too, which is the
+ opposite of what this boundary is for.
+ """
+ try:
+ if not isinstance(candidate, PurgeEntityPolicy):
+ logger.error(
+ "purge_policy: %s returned a %s; expected PurgeEntityPolicy",
+ HOST_POLICIES_CONFIG_KEY,
+ type(candidate).__name__,
+ )
+ return None
+ if not isinstance(candidate.model, type):
+ # Everything downstream treats the root as a mapped class:
+ # sa.inspect, the registry index, the duplicate count.
+ logger.error(
+ "purge_policy: host policy root %s is not a class",
+ _policy_label(candidate),
+ )
+ return None
+ if candidate.model in builtin_roots:
+ logger.error(
+ "purge_policy: host policy for built-in root %s ignored",
+ candidate.model.__name__,
+ )
+ return None
+ if candidate.entity_type in builtin_types:
+ # Core branches on entity_type -- the dataset impact path, the tag
+ # object type, the dataset permission name -- so a host reusing
+ # one of its names would have that logic pointed at its own rows.
+ logger.error(
+ "purge_policy: host policy for %s claims the reserved entity type %r",
+ candidate.model.__name__,
+ candidate.entity_type,
+ )
+ return None
+ # Validated first, so a table the metadata simply does not contain is
+ # reported as such rather than as an ambiguous name.
+ unusable: set[str] = {
+ dependency.classification.value
+ for dependency in candidate.dependencies
+ if dependency.classification
+ in {
+ DependencyClassification.BLOCK,
+ DependencyClassification.LISTENER_EFFECT,
+ }
+ }
+ if unusable:
+ # Both resolve from core's own identities: the stock listener
+ # actions decide what to delete from a built-in entity type, and
+ # blockers are applied by the stock validator. Declared by a host,
+ # they are accepted and then never carried out -- silence rather
+ # than a failure, which is the one outcome worth refusing. A host
+ # blocks by raising PurgeBlockedError from its own validate.
+ raise RuntimeError(
+ f"{', '.join(sorted(unusable))} dependencies cannot be carried "
+ "out for a host root"
+ )
+ _validated_policy(candidate)
+ conflict: str | None = _unqualified_table_conflict(candidate)
+ if conflict is not None:
+ raise RuntimeError(
+ f"table {conflict!r} cannot be identified unambiguously by "
+ "name; schema-qualified roots are not supported"
+ )
+ except Exception as ex: # pylint: disable=broad-except
+ # Validated at admission rather than when a row is purged: a
+ # declaration the cleanup cannot execute would otherwise fail once per
+ # eligible row, counted as a cascade failure, and stay invisible until
+ # something aged past the window.
+ logger.error(
+ "purge_policy: host policy for %s rejected: %s",
+ _policy_label(candidate),
+ ex,
+ )
+ return None
+ return candidate
+
+
+def _host_purge_policies(
+ provider: Callable[[], Any] | None,
+) -> tuple[PurgeEntityPolicy, ...] | None:
+ """Return the purge policies a host installed for its own roots.
+
+ A host distribution can carry ``SoftDeleteMixin`` entities this package
+ cannot import. The retention task discovers those entities through the
+ mixin registry, so without a policy they reach the cascade as an
+ unsupported model. The host therefore declares their purge behavior and
+ installs it under ``PURGE_POLICIES_FUNC``.
+
+ Host boundary: everything about a host policy is settled here, where it is
+ admitted -- an unavailable provider, a payload that is not an iterable of
+ policies, a root that is not a class, a collision with a root or entity
+ type declared in this package, two declarations for one root, and a
+ declaration the shared cleanup could not execute. Each is logged and
+ dropped, so that root is reported as unsupported instead of failing row by
+ row in a scheduled run. A broken host declaration must not take the purge
+ down with it, and must never redefine how a chart, dashboard or dataset is
+ purged.
+
+ Returns ``None`` for the one failure that may be transient -- the provider
+ itself could not be called. The caller then serves the built-in roots
+ *without* publishing, so the next resolution tries again instead of
+ freezing a bad moment in for the life of the process. A payload that is
+ merely malformed is a host bug that will not fix itself, so it is
+ published as an empty result rather than retried on every read.
+ """
+ if provider is None:
+ return ()
+ try:
+ provided: Any = provider()
+ except Exception: # pylint: disable=broad-except
+ logger.exception(
+ "purge_policy: %s is unavailable; keeping built-in roots only",
+ HOST_POLICIES_CONFIG_KEY,
+ )
+ return None
+ candidates: tuple[Any, ...] | None = _host_policy_payload(provided)
+ if candidates is None:
+ return ()
+
+ builtin_policies: tuple[PurgeEntityPolicy, ...] = _builtin_purge_policies()
+ builtin_roots: frozenset[type[Any]] = frozenset(
+ policy.model for policy in builtin_policies
+ )
+ builtin_types: frozenset[str] = frozenset(
+ policy.entity_type for policy in builtin_policies
+ )
+ admitted: list[PurgeEntityPolicy] = [
+ policy
+ for candidate in candidates
+ if (policy := _admitted_host_policy(candidate, builtin_roots, builtin_types))
+ is not None
+ ]
+
+ # Counted after admission, where every root is known to be a class and so
+ # hashable. Which of two declarations for one root is authoritative is
+ # undecidable, and the loser would still delete rows, so both are dropped:
+ # the root is reported as unsupported, which is the recoverable outcome.
+ # It also keeps the duplicate away from validate_unique_root_policies,
+ # whose ValueError would abort the whole scheduled run.
+ counts: dict[type[Any], int] = {}
+ for policy in admitted:
+ counts[policy.model] = counts.get(policy.model, 0) + 1
+ for model, count in counts.items():
+ if count > 1:
+ logger.error(
+ "purge_policy: %s declares %d policies for %s; ignoring all of them",
+ HOST_POLICIES_CONFIG_KEY,
+ count,
+ model.__name__,
+ )
+ return tuple(policy for policy in admitted if counts[policy.model] == 1)
+
+
+#: Sentinel distinguishing "no index resolved yet" from an index resolved for
+#: no provider, which is the ordinary case.
+_UNRESOLVED: Any = object()
+
+
+@dataclass
+class _ResolvedRegistry:
+ """The index built for one provider, with the roots validated so far.
+
+ Compared by provider *identity* rather than memoized with ``lru_cache``: a
+ cache keyed on the provider would hash it, and a host callable
+ implemented as a mutable dataclass instance is unhashable. That would
+ raise here -- before the host boundary could isolate the host's mistake --
+ and stop the built-in roots purging too.
+ """
+
+ provider: Any = _UNRESOLVED
+ registry: Mapping[type[Any], PurgeEntityPolicy] = field(
+ default_factory=lambda: MappingProxyType({})
+ )
+ validated: set[type[Any]] = field(default_factory=set)
+
+
+_RESOLVED: _ResolvedRegistry = _ResolvedRegistry()
+
+#: Per-thread marker for "this thread is already resolving". A host provider
+#: may build its policy from a built-in one -- ``replace(get_purge_policy(
+#: Slice), model=HostRoot, ...)`` is the obvious way to borrow the stock
+#: callbacks -- which re-enters resolution while the first call is still in
+#: the provider. The nested call is handed the state already published
+#: instead, which always carries the built-in roots.
+_RESOLVING: threading.local = threading.local()
+
+
+def _resolved_registry() -> _ResolvedRegistry:
+ """Return the state resolved for the installed provider, building it once.
+
+ Readers take a snapshot and a rebuild publishes a new one, so a caller
+ always sees an index and its validation record together. Publishing is a
+ single rebind, which is what makes it safe to do this without a lock: two
+ concurrent first uses may each invoke the provider and each publish, but
+ the two snapshots are equivalent and neither is ever seen half-built.
+ Paying for a duplicate provider call is the deliberate trade -- a lock
+ here deadlocks the moment a provider resolves a built-in policy of its
+ own, and a hung purge is worse than a repeated call.
+ """
+ provider: Callable[[], Any] | None = (
+ current_app.config.get(HOST_POLICIES_CONFIG_KEY) if has_app_context() else None
+ )
+ resolved: _ResolvedRegistry = _RESOLVED
+ if resolved.provider is provider:
Review Comment:
**major**: Cache identity is the provider object, so a provider that returns a fresh callable (a lambda built per call, say) or a config toggled between app contexts rebuilds on every call. In a multi-app process it also thrashes the single global snapshot, and the provider then runs once per root.
##########
superset/commands/deletion_retention/purge_policy.py:
##########
@@ -860,16 +913,349 @@ def declare(
cleanup_permission=cleanup_dataset_permission,
),
}
- return validate_unique_root_policies(registry.values())
+ return tuple(registry.values())
+
+
+def _policy_label(policy: Any) -> str:
+ """A log-safe name for a payload whose shape is not yet established."""
+ model: Any = getattr(policy, "model", None)
+ return str(getattr(model, "__name__", type(model).__name__))
+
+def _host_policy_payload(provided: Any) -> tuple[Any, ...] | None:
+ """Materialize the provider's payload, or reject it.
-@lru_cache(maxsize=None)
-def _validated_purge_policy(model: type[Any]) -> PurgeEntityPolicy:
- """Validate and return one root policy without blocking unrelated roots."""
+ Any iterable is accepted -- a list, a tuple, ``dict.values()``, a
+ generator -- since the documented contract is a sequence of policies, not
+ one particular container. Strings and bytes are excluded because they
+ iterate into characters, which would read as a sequence of bad members
+ rather than the wrong type. Materialized once: a generator cannot be
+ walked twice.
+ """
+ if isinstance(provided, (str, bytes)) or not isinstance(provided, Iterable):
+ logger.error(
+ "purge_policy: %s returned %s; expected an iterable of PurgeEntityPolicy",
+ HOST_POLICIES_CONFIG_KEY,
+ type(provided).__name__,
+ )
+ return None
try:
- policy: PurgeEntityPolicy = purge_policy_registry()[model]
- except KeyError as ex:
- raise ValueError(f"Unsupported purge model: {model.__name__}") from ex
+ return tuple(provided)
+ except Exception: # pylint: disable=broad-except
+ logger.exception(
+ "purge_policy: %s could not be read; keeping built-in roots only",
+ HOST_POLICIES_CONFIG_KEY,
+ )
+ return None
+
+
+def _unqualified_table_conflict(policy: PurgeEntityPolicy) -> str | None:
+ """Return the first table a bare-name lookup would resolve wrongly.
+
+ Discovery records table identities by bare name while a ``MetaData`` keys
+ them by schema, so a declared ``child`` resolves to the default-schema
+ table even where the policy meant ``host.child`` -- and the cleanup would
+ delete rows belonging to unrelated roots. Carrying schema-qualified
+ identities through discovery and execution is the better fix; until then
+ the ambiguous shape is refused rather than silently mis-resolved.
+ """
+ root_table: sa.Table = sa.inspect(policy.model).local_table
+ metadata: sa.MetaData = root_table.metadata
+ keys_by_name: dict[str, list[str]] = {}
+ for table in metadata.tables.values():
+ keys_by_name.setdefault(table.name, []).append(table.key)
+ declared: set[str] = {root_table.name} | {
+ dependency.key.related_table
+ for dependency in policy.dependencies
+ if dependency.classification in _EXECUTABLE_CLASSIFICATIONS
+ }
+ for name in sorted(declared):
+ keys: list[str] = keys_by_name.get(name, [])
+ if len(keys) != 1 or keys[0] != name:
+ return name
+ return None
+
+
+def _admitted_host_policy(
+ candidate: Any, builtin_roots: frozenset[type[Any]], builtin_types: frozenset[str]
+) -> PurgeEntityPolicy | None:
+ """Return *candidate* if it is a usable host policy, else ``None``.
+
+ Every check runs behind one handler rather than guarding each attribute:
+ the payload is host-supplied, so reading ``model`` or hashing it may
+ itself raise -- and an exception escaping here would break
+ ``purge_policy_registry`` for the built-in roots too, which is the
+ opposite of what this boundary is for.
+ """
+ try:
+ if not isinstance(candidate, PurgeEntityPolicy):
+ logger.error(
+ "purge_policy: %s returned a %s; expected PurgeEntityPolicy",
+ HOST_POLICIES_CONFIG_KEY,
+ type(candidate).__name__,
+ )
+ return None
+ if not isinstance(candidate.model, type):
+ # Everything downstream treats the root as a mapped class:
+ # sa.inspect, the registry index, the duplicate count.
+ logger.error(
+ "purge_policy: host policy root %s is not a class",
+ _policy_label(candidate),
+ )
+ return None
+ if candidate.model in builtin_roots:
+ logger.error(
+ "purge_policy: host policy for built-in root %s ignored",
+ candidate.model.__name__,
+ )
+ return None
+ if candidate.entity_type in builtin_types:
+ # Core branches on entity_type -- the dataset impact path, the tag
+ # object type, the dataset permission name -- so a host reusing
+ # one of its names would have that logic pointed at its own rows.
+ logger.error(
+ "purge_policy: host policy for %s claims the reserved entity type %r",
+ candidate.model.__name__,
+ candidate.entity_type,
+ )
+ return None
+ # Validated first, so a table the metadata simply does not contain is
+ # reported as such rather than as an ambiguous name.
+ unusable: set[str] = {
+ dependency.classification.value
+ for dependency in candidate.dependencies
+ if dependency.classification
+ in {
+ DependencyClassification.BLOCK,
+ DependencyClassification.LISTENER_EFFECT,
+ }
+ }
+ if unusable:
+ # Both resolve from core's own identities: the stock listener
+ # actions decide what to delete from a built-in entity type, and
+ # blockers are applied by the stock validator. Declared by a host,
+ # they are accepted and then never carried out -- silence rather
+ # than a failure, which is the one outcome worth refusing. A host
+ # blocks by raising PurgeBlockedError from its own validate.
+ raise RuntimeError(
+ f"{', '.join(sorted(unusable))} dependencies cannot be carried "
+ "out for a host root"
+ )
+ _validated_policy(candidate)
+ conflict: str | None = _unqualified_table_conflict(candidate)
+ if conflict is not None:
+ raise RuntimeError(
+ f"table {conflict!r} cannot be identified unambiguously by "
+ "name; schema-qualified roots are not supported"
+ )
+ except Exception as ex: # pylint: disable=broad-except
+ # Validated at admission rather than when a row is purged: a
+ # declaration the cleanup cannot execute would otherwise fail once per
+ # eligible row, counted as a cascade failure, and stay invisible until
+ # something aged past the window.
+ logger.error(
+ "purge_policy: host policy for %s rejected: %s",
+ _policy_label(candidate),
+ ex,
+ )
+ return None
+ return candidate
+
+
+def _host_purge_policies(
+ provider: Callable[[], Any] | None,
+) -> tuple[PurgeEntityPolicy, ...] | None:
+ """Return the purge policies a host installed for its own roots.
+
+ A host distribution can carry ``SoftDeleteMixin`` entities this package
+ cannot import. The retention task discovers those entities through the
+ mixin registry, so without a policy they reach the cascade as an
+ unsupported model. The host therefore declares their purge behavior and
+ installs it under ``PURGE_POLICIES_FUNC``.
+
+ Host boundary: everything about a host policy is settled here, where it is
+ admitted -- an unavailable provider, a payload that is not an iterable of
+ policies, a root that is not a class, a collision with a root or entity
+ type declared in this package, two declarations for one root, and a
+ declaration the shared cleanup could not execute. Each is logged and
+ dropped, so that root is reported as unsupported instead of failing row by
+ row in a scheduled run. A broken host declaration must not take the purge
+ down with it, and must never redefine how a chart, dashboard or dataset is
+ purged.
+
+ Returns ``None`` for the one failure that may be transient -- the provider
+ itself could not be called. The caller then serves the built-in roots
+ *without* publishing, so the next resolution tries again instead of
+ freezing a bad moment in for the life of the process. A payload that is
+ merely malformed is a host bug that will not fix itself, so it is
+ published as an empty result rather than retried on every read.
+ """
+ if provider is None:
+ return ()
+ try:
+ provided: Any = provider()
+ except Exception: # pylint: disable=broad-except
+ logger.exception(
+ "purge_policy: %s is unavailable; keeping built-in roots only",
+ HOST_POLICIES_CONFIG_KEY,
+ )
+ return None
+ candidates: tuple[Any, ...] | None = _host_policy_payload(provided)
+ if candidates is None:
+ return ()
+
+ builtin_policies: tuple[PurgeEntityPolicy, ...] = _builtin_purge_policies()
+ builtin_roots: frozenset[type[Any]] = frozenset(
+ policy.model for policy in builtin_policies
+ )
+ builtin_types: frozenset[str] = frozenset(
+ policy.entity_type for policy in builtin_policies
+ )
+ admitted: list[PurgeEntityPolicy] = [
+ policy
+ for candidate in candidates
+ if (policy := _admitted_host_policy(candidate, builtin_roots, builtin_types))
+ is not None
+ ]
+
+ # Counted after admission, where every root is known to be a class and so
+ # hashable. Which of two declarations for one root is authoritative is
+ # undecidable, and the loser would still delete rows, so both are dropped:
+ # the root is reported as unsupported, which is the recoverable outcome.
+ # It also keeps the duplicate away from validate_unique_root_policies,
+ # whose ValueError would abort the whole scheduled run.
+ counts: dict[type[Any], int] = {}
+ for policy in admitted:
+ counts[policy.model] = counts.get(policy.model, 0) + 1
+ for model, count in counts.items():
+ if count > 1:
+ logger.error(
+ "purge_policy: %s declares %d policies for %s; ignoring all of them",
+ HOST_POLICIES_CONFIG_KEY,
+ count,
+ model.__name__,
+ )
+ return tuple(policy for policy in admitted if counts[policy.model] == 1)
+
+
+#: Sentinel distinguishing "no index resolved yet" from an index resolved for
+#: no provider, which is the ordinary case.
+_UNRESOLVED: Any = object()
+
+
+@dataclass
+class _ResolvedRegistry:
+ """The index built for one provider, with the roots validated so far.
+
+ Compared by provider *identity* rather than memoized with ``lru_cache``: a
+ cache keyed on the provider would hash it, and a host callable
+ implemented as a mutable dataclass instance is unhashable. That would
+ raise here -- before the host boundary could isolate the host's mistake --
+ and stop the built-in roots purging too.
+ """
+
+ provider: Any = _UNRESOLVED
+ registry: Mapping[type[Any], PurgeEntityPolicy] = field(
+ default_factory=lambda: MappingProxyType({})
+ )
+ validated: set[type[Any]] = field(default_factory=set)
+
+
+_RESOLVED: _ResolvedRegistry = _ResolvedRegistry()
+
+#: Per-thread marker for "this thread is already resolving". A host provider
+#: may build its policy from a built-in one -- ``replace(get_purge_policy(
+#: Slice), model=HostRoot, ...)`` is the obvious way to borrow the stock
+#: callbacks -- which re-enters resolution while the first call is still in
+#: the provider. The nested call is handed the state already published
+#: instead, which always carries the built-in roots.
+_RESOLVING: threading.local = threading.local()
+
+
+def _resolved_registry() -> _ResolvedRegistry:
+ """Return the state resolved for the installed provider, building it once.
+
+ Readers take a snapshot and a rebuild publishes a new one, so a caller
+ always sees an index and its validation record together. Publishing is a
+ single rebind, which is what makes it safe to do this without a lock: two
+ concurrent first uses may each invoke the provider and each publish, but
+ the two snapshots are equivalent and neither is ever seen half-built.
+ Paying for a duplicate provider call is the deliberate trade -- a lock
+ here deadlocks the moment a provider resolves a built-in policy of its
+ own, and a hung purge is worse than a repeated call.
+ """
+ provider: Callable[[], Any] | None = (
+ current_app.config.get(HOST_POLICIES_CONFIG_KEY) if has_app_context() else None
+ )
+ resolved: _ResolvedRegistry = _RESOLVED
+ if resolved.provider is provider:
+ return resolved
+ if getattr(_RESOLVING, "active", False):
+ # A nested resolution: the provider is building its policy from a
+ # built-in one. It needs the roots this package declares -- not
+ # whatever happens to be published, which on a first resolution is
+ # still the empty initial snapshot.
+ return _builtin_registry()
+ _RESOLVING.active = True
+ try:
+ host_policies: tuple[PurgeEntityPolicy, ...] | None = _host_purge_policies(
+ provider
+ )
+ finally:
+ _RESOLVING.active = False
+ if host_policies is None:
+ # The provider could not be called. Serve the built-in roots without
+ # publishing, so a failure that turns out to be transient is retried
+ # on the next resolution rather than fixed in place.
+ return _builtin_registry()
+ rebuilt = _ResolvedRegistry(
+ provider=provider,
+ registry=validate_unique_root_policies(
+ (*_builtin_purge_policies(), *host_policies)
+ ),
+ # Host roots were validated as they were admitted. Built-in roots stay
+ # lazy, so one unusable built-in cannot block the others.
+ validated={policy.model for policy in host_policies},
+ )
+ globals()["_RESOLVED"] = rebuilt
Review Comment:
**major**: `globals()["_RESOLVED"] = rebuilt` is published with no lock, so two threads can race. The slower one overwrites the newer snapshot, and the `validated` set is mutated in `get_purge_policy` without synchronization.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(retention): let a host install purge policies for its own soft-delete roots [superset]
Posted by "msyavuz (via GitHub)" <gi...@apache.org>.
msyavuz commented on code in PR #44892:
URL: https://github.com/apache/superset/pull/44892#discussion_r4183383151
##########
superset/commands/deletion_retention/purge_policy.py:
##########
@@ -860,7 +894,96 @@ def declare(
cleanup_permission=cleanup_dataset_permission,
),
}
- return validate_unique_root_policies(registry.values())
+ return tuple(registry.values())
+
+
+def _host_purge_policies() -> tuple[PurgeEntityPolicy, ...]:
+ """Return the purge policies a host installed for its own roots.
+
+ A host distribution can carry ``SoftDeleteMixin`` entities this package
+ cannot import. The retention task discovers those entities through the
+ mixin registry, so without a policy they reach the cascade as an
+ unsupported model. The host therefore declares their purge behavior and
+ installs it under ``PURGE_POLICIES_FUNC``.
+
+ Host boundary: an unavailable provider, a malformed payload, a policy that
+ collides with a root declared here, and two policies for one host root are
+ each logged and dropped. A broken host declaration must not take the
+ scheduled purge down with it, and must never redefine how a chart,
+ dashboard or dataset is purged.
+ """
+ if not has_app_context():
+ return ()
+ provider: Callable[[], Any] | None = current_app.config.get(
+ HOST_POLICIES_CONFIG_KEY
+ )
+ if provider is None:
+ return ()
+ try:
+ provided: Any = provider()
+ except Exception: # pylint: disable=broad-except
+ logger.exception(
+ "purge_policy: %s is unavailable; keeping built-in roots only",
+ HOST_POLICIES_CONFIG_KEY,
+ )
+ return ()
+ if not isinstance(provided, (list, tuple)) or not all(
+ isinstance(policy, PurgeEntityPolicy) for policy in provided
+ ):
+ logger.error(
+ "purge_policy: %s returned %s; expected a sequence of PurgeEntityPolicy",
+ HOST_POLICIES_CONFIG_KEY,
+ type(provided).__name__,
+ )
+ return ()
+ builtin_roots: frozenset[type[Any]] = frozenset(
+ policy.model for policy in _builtin_purge_policies()
+ )
+ declared: dict[type[Any], int] = {}
+ for policy in provided:
+ declared[policy.model] = declared.get(policy.model, 0) + 1
+ # Which of two declarations for one root is authoritative is undecidable,
+ # and the loser would still delete rows. Dropping both leaves the model
+ # reported as unsupported, which is the recoverable outcome. Resolving it
+ # here also keeps the duplicate away from validate_unique_root_policies,
+ # whose ValueError would abort the whole scheduled run.
+ duplicated: list[type[Any]] = [
+ model for model, count in declared.items() if count > 1
+ ]
+ for model in duplicated:
+ logger.error(
+ "purge_policy: %s declares %d policies for %s; ignoring all of them",
+ HOST_POLICIES_CONFIG_KEY,
+ declared[model],
+ model.__name__,
+ )
+ accepted: list[PurgeEntityPolicy] = []
+ for policy in provided:
+ if policy.model in builtin_roots:
+ logger.error(
+ "purge_policy: host policy for built-in root %s ignored",
+ policy.model.__name__,
+ )
+ continue
+ if policy.model in duplicated:
+ continue
+ accepted.append(policy)
+ return tuple(accepted)
+
+
+def purge_policy_registry() -> Mapping[type[Any], PurgeEntityPolicy]:
+ """Index the built-in purge roots plus any the host installed.
+
+ Uncached on purpose, unlike its two inputs: the built-in declarations are
+ built once per process, while a host policy is resolved per call, so a
+ provider installed after the first purge is honored for any root not yet
+ resolved. ``get_purge_policy`` memoizes per model, so replacing the policy
+ of a root it has already resolved needs a restart -- and that memoization
+ is why rebuilding this small index is off the hot path.
+ """
+ return validate_unique_root_policies(
Review Comment:
**major**: purge_policy_registry() is no longer cached and now re-runs provider() and validate_unique_root_policies on every call, and `_purge_roots` calls it once per model. A slow or non-deterministic provider is invoked repeatedly, and the returned policy objects can differ between calls.
##########
superset/commands/deletion_retention/purge_policy.py:
##########
@@ -860,7 +894,96 @@ def declare(
cleanup_permission=cleanup_dataset_permission,
),
}
- return validate_unique_root_policies(registry.values())
+ return tuple(registry.values())
+
+
+def _host_purge_policies() -> tuple[PurgeEntityPolicy, ...]:
+ """Return the purge policies a host installed for its own roots.
+
+ A host distribution can carry ``SoftDeleteMixin`` entities this package
+ cannot import. The retention task discovers those entities through the
+ mixin registry, so without a policy they reach the cascade as an
+ unsupported model. The host therefore declares their purge behavior and
+ installs it under ``PURGE_POLICIES_FUNC``.
+
+ Host boundary: an unavailable provider, a malformed payload, a policy that
+ collides with a root declared here, and two policies for one host root are
+ each logged and dropped. A broken host declaration must not take the
+ scheduled purge down with it, and must never redefine how a chart,
+ dashboard or dataset is purged.
+ """
+ if not has_app_context():
Review Comment:
**major**: Outside an app context, host roots silently drop out and then get reported as unsupported. Because `get_purge_policy` memoizes, a failure from a context-less first call may be cached and persist until restart.
##########
superset/tasks/deletion_retention.py:
##########
@@ -132,6 +133,54 @@ def _report_model_counts(outcome: str, counts: dict[str, int]) -> None:
)
+@dataclass
+class _PassTotals:
+ """What one pass over the soft-delete roots produced."""
+
+ purged: dict[str, int] = field(default_factory=dict)
+ would_purge: dict[str, int] = field(default_factory=dict)
+ unsupported: dict[str, int] = field(default_factory=dict)
+ cascade_failures: int = 0
+ blocked: int = 0
+ scan_failures: int = 0
+
+
+def _purge_roots(cutoff: datetime, dry_run: bool) -> _PassTotals:
+ """Process each registered root, isolating one root's failure from the rest."""
+ totals = _PassTotals()
+ for model in _soft_delete_models():
+ entity_type = _model_table_name(model)
+ try:
+ if model not in purge_policy_registry():
+ totals.unsupported[entity_type] = 1
+ logger.warning(
+ "deletion_retention: skipping %s: no purge policy", entity_type
+ )
+ stats_logger_manager.instance.incr(
+ f"{_METRIC_PREFIX}.unsupported_models.{entity_type}"
+ )
+ continue
+ purged_n, would_n, failed_n, blocked_n = _purge_model(
+ model, cutoff, dry_run
+ )
+ except Exception: # pylint: disable=broad-except
Review Comment:
**major**: The broad except now also swallows a RuntimeError from policy validation (`Incomplete purge policy`) raised via `purge_policy_registry` or `get_purge_policy`. A bad host policy then shows up only as `scan_failures` on every run, and it is mislabeled as a scan error.
##########
superset/commands/deletion_retention/purge_policy.py:
##########
@@ -909,23 +1032,211 @@ def _validated_purge_policy(model: type[Any]) -> PurgeEntityPolicy:
return policy
+def _ownership_dependency(
+ policy: PurgeEntityPolicy, related_table: str
+) -> DependencyPolicy | None:
+ """The single owned/association edge attaching *related_table*, if clear."""
+ candidates: tuple[DependencyPolicy, ...] = tuple(
+ dependency
+ for dependency in policy.dependencies
+ if dependency.classification
+ in {DependencyClassification.OWNED, DependencyClassification.ASSOCIATION}
+ and dependency.key.related_table == related_table
+ and dependency.key.direction == "inbound"
+ )
+ return candidates[0] if len(candidates) == 1 else None
+
+
+def _validate_owned_traversal(policy: PurgeEntityPolicy) -> None:
+ """Reject an owned table reachable only through an association.
+
+ ``cascade_hard_delete`` empties associations before owned children, while
+ an owned table's predicate selects its rows *through* its ownership path
+ (see ``_owner_value_select``). If a hop on that path is an association,
+ its rows are already gone when the owned delete runs: the statement
+ matches nothing, leaving the descendants orphaned where foreign keys are
+ unenforced and blocking the root's delete where they are not.
+
+ Refused at declaration time rather than executed. The shape has a
+ remedy -- classify the intermediate table as owned, which places it in
+ the same phase as what it leads to.
+ """
+ root_table: str = sa.inspect(policy.model).local_table.name
+ for dependency in policy.dependencies:
+ if dependency.classification is not DependencyClassification.OWNED:
+ continue
+ table_name: str = dependency.key.owner_table
+ visited: set[str] = set()
+ while table_name != root_table and table_name not in visited:
+ visited.add(table_name)
+ hop: DependencyPolicy | None = _ownership_dependency(policy, table_name)
+ if hop is None:
+ # An absent or ambiguous path is reported by coverage and by
+ # _ownership_edge at execution; not this check's business.
+ break
+ if hop.classification is DependencyClassification.ASSOCIATION:
+ raise RuntimeError(
+ f"Owned dependency {dependency.key.describe()} is reachable "
+ f"only through association {hop.key.describe()}; "
+ "associations are deleted first, so the owned rows would "
+ "be orphaned"
+ )
+ table_name = hop.key.owner_table
+
+
+def _tables_share_a_foreign_key(metadata: sa.MetaData, first: str, second: str) -> bool:
Review Comment:
**minor**: `_tables_share_a_foreign_key` only compares the first element of each foreign key constraint and only checks direct FKs. Cross-references between owned siblings through a third table or a multi-column FK are not detected, so the sibling-order check can miss real cases.
##########
superset/commands/deletion_retention/purge_policy.py:
##########
@@ -329,34 +363,34 @@ def _discover_table_dependencies(
if constraint.elements[0].column.table is table:
discovered.add(_fk_key(table, constraint, "inbound"))
discovered.update(
- _discover_recursive_table_dependencies(
- table,
- recursive_tables=recursive_tables,
- visited=next_visited,
- )
+ _discover_recursive_table_dependencies(table, recursive_tables, seen)
)
return frozenset(discovered)
def _discover_recursive_table_dependencies(
table: sa.Table,
- *,
recursive_tables: frozenset[str],
- visited: frozenset[str],
+ seen: _Traversal,
) -> frozenset[DependencyKey]:
- """Discover dependencies for named recursive tables not already visited."""
+ """Discover dependencies for named recursive tables not already visited.
+
+ Sorted so a traversal is reproducible, and driven off the shared
+ bookkeeping so each table is walked at most once per kind of walk.
+ """
discovered: set[DependencyKey] = set()
- for recursive_table_name in recursive_tables - visited:
+ for recursive_table_name in sorted(recursive_tables - seen.tables):
+ # Re-checked inside the loop: the set grows as the walk descends, so
+ # a table a nested call already covered is skipped rather than
+ # re-entered to return nothing.
+ if recursive_table_name in seen.tables:
Review Comment:
**minor**: The shared `seen` set changes discovery output for graphs where a table is reached via different paths. The `_Traversal` split of tables and mappers was added to mitigate this, but I only see tests for the call count, not for built-in key-set equivalence.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(retention): let a host install purge policies for its own soft-delete roots [superset]
Posted by "codeant-ai-for-open-source[bot] (via GitHub)" <gi...@apache.org>.
codeant-ai-for-open-source[bot] commented on PR #44892:
URL: https://github.com/apache/superset/pull/44892#issuecomment-6014480062
## CodeAnt PR Risk: Low Risk
- The PR appears safe to merge; malformed host policies are rejected without disrupting the built-in purge policies.
- Per-root scan failures are isolated and reported while counts for previously committed deletions are preserved.
- Added tests cover provider failures, duplicate policies, policy validation, dependency traversal, and partial-scan failures.
**Assessed commit:** `4da324808044`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(retention): let a host install purge policies for its own soft-delete roots [superset]
Posted by "sadpandajoe (via GitHub)" <gi...@apache.org>.
sadpandajoe commented on code in PR #44892:
URL: https://github.com/apache/superset/pull/44892#discussion_r4186369948
##########
superset/commands/deletion_retention/purge_policy.py:
##########
@@ -905,32 +1080,311 @@ def _validated_purge_policy(model: type[Any]) -> PurgeEntityPolicy:
details = (
f"{details}; " if details else ""
) + f"stale_listeners=[{', '.join(coverage.stale_listeners)}]"
- raise RuntimeError(f"Incomplete purge policy for {model.__name__}: {details}")
+ raise RuntimeError(
+ f"Incomplete purge policy for {policy.model.__name__}: {details}"
+ )
return policy
-def _validate_executable_declarations(policy: PurgeEntityPolicy) -> None:
- """Reject executable classifications missing their required action metadata."""
+def _ownership_dependency(
+ policy: PurgeEntityPolicy, related_table: str
+) -> DependencyPolicy | None:
+ """The single owned/association edge attaching *related_table*, if clear."""
+ candidates: tuple[DependencyPolicy, ...] = tuple(
+ dependency
+ for dependency in policy.dependencies
+ if dependency.classification
+ in {DependencyClassification.OWNED, DependencyClassification.ASSOCIATION}
+ and dependency.key.related_table == related_table
+ and dependency.key.direction == "inbound"
+ )
+ return candidates[0] if len(candidates) == 1 else None
+
+
+def _validate_owned_traversal(policy: PurgeEntityPolicy) -> None:
+ """Reject an owned table reachable only through an association.
+
+ ``cascade_hard_delete`` empties associations before owned children, while
+ an owned table's predicate selects its rows *through* its ownership path
+ (see ``_owner_value_select``). If a hop on that path is an association,
+ its rows are already gone when the owned delete runs: the statement
+ matches nothing, leaving the descendants orphaned where foreign keys are
+ unenforced and blocking the root's delete where they are not.
+
+ Refused at declaration time rather than executed. The shape has a
+ remedy -- classify the intermediate table as owned, which places it in
+ the same phase as what it leads to.
+ """
+ root_table: str = sa.inspect(policy.model).local_table.name
for dependency in policy.dependencies:
- if (
- dependency.classification is DependencyClassification.LISTENER_EFFECT
- and dependency.listener_action is None
- ):
+ if dependency.classification is not DependencyClassification.OWNED:
+ continue
+ table_name: str = dependency.key.owner_table
+ visited: set[str] = set()
+ while table_name != root_table and table_name not in visited:
+ visited.add(table_name)
+ hop: DependencyPolicy | None = _ownership_dependency(policy, table_name)
+ if hop is None:
+ # An absent or ambiguous path is reported by coverage and by
+ # _ownership_edge at execution; not this check's business.
+ break
+ if hop.classification is DependencyClassification.ASSOCIATION:
+ raise RuntimeError(
+ f"Owned dependency {dependency.key.describe()} is reachable "
+ f"only through association {hop.key.describe()}; "
+ "associations are deleted first, so the owned rows would "
+ "be orphaned"
+ )
+ table_name = hop.key.owner_table
+
+
+def _tables_share_a_foreign_key(metadata: sa.MetaData, first: str, second: str) -> bool:
+ """Whether either table declares a foreign key into the other."""
+ for owner_name, other_name in ((first, second), (second, first)):
+ table: sa.Table | None = metadata.tables.get(owner_name)
+ if table is None:
+ continue
+ for constraint in table.foreign_key_constraints:
+ if constraint.elements[0].column.table.name == other_name:
+ return True
+ return False
+
+
+def _validate_scanner_requirements(policy: PurgeEntityPolicy) -> None:
+ """Reject a root the scheduled scan cannot page through.
+
+ The retention task selects, windows and orders eligible rows by ``id``,
+ and the cascade pins each row by it. A root keyed on something else -- a
+ UUID primary key with no ``id`` column -- raises inside the scan, outside
+ the per-row error handling, so the run aborts before the remaining roots
+ are reached.
+ """
+ table: sa.Table = sa.inspect(policy.model).local_table
+ if "id" not in table.c:
+ raise RuntimeError(
+ f"Purge root {policy.model.__name__} has no 'id' column; the "
+ "scheduled scan pages eligible rows by id"
+ )
+ if "deleted_at" not in table.c:
+ raise RuntimeError(
+ f"Purge root {policy.model.__name__} has no 'deleted_at' column; "
+ "every purge path requires the row to be archived first"
+ )
+
+
+def _validate_recursive_ownership(policy: PurgeEntityPolicy) -> None:
+ """Reject a self-referencing owned table under the stock cleanup.
+
+ ``delete_owned_children`` issues one statement per declared edge, so a
+ table that owns itself is pruned one level deep: a three-level tree
+ leaves its deepest rows behind -- orphaned where foreign keys are
+ unenforced, and blocking the root's own delete where they are not. A host
+ declaring this shape supplies cleanup that walks the whole sub-tree.
+ """
+ for dependency in policy.dependencies:
+ if dependency.classification is not DependencyClassification.OWNED:
+ continue
+ key: DependencyKey = dependency.key
+ if key.owner_table != key.related_table:
+ continue
+ if policy.delete_owned_children is delete_owned_children:
raise RuntimeError(
- f"Missing listener action for {dependency.key.describe()}"
+ f"Owned dependency {key.describe()} is self-referencing; the "
+ "stock owned-child cleanup deletes one level, so this policy "
+ "must supply its own delete_owned_children"
)
- if (
- dependency.classification is DependencyClassification.VERSION_OWNED
- and dependency.version_column is None
- ):
+
+
+def _validate_sibling_ownership(policy: PurgeEntityPolicy) -> None:
+ """Reject owned siblings that reference one another.
+
+ Owned deletes are sorted by ownership depth with a stable sort, so two
+ owned tables at the same depth are deleted in the order they were
+ declared. With a foreign key between them, that order decides the
+ outcome: removing the referenced table first is refused by an enforced
+ constraint and rolls the purge back. Ordering deletes by their own
+ foreign keys would be the richer fix; refusing the shape keeps the
+ engine's contract honest until something needs it.
+ """
+ metadata: sa.MetaData = sa.inspect(policy.model).local_table.metadata
+ by_depth: dict[int, list[DependencyPolicy]] = {}
+ for dependency in policy.dependencies:
+ if dependency.classification is not DependencyClassification.OWNED:
+ continue
+ depth: int = _dependency_owner_depth(policy, dependency.key)
+ by_depth.setdefault(depth, []).append(dependency)
+ for siblings in by_depth.values():
+ for index, first in enumerate(siblings):
+ for second in siblings[index + 1 :]:
+ if first.key.related_table == second.key.related_table:
+ continue
+ if _tables_share_a_foreign_key(
+ metadata, first.key.related_table, second.key.related_table
+ ):
+ raise RuntimeError(
+ f"Owned dependencies {first.key.describe()} and "
+ f"{second.key.describe()} reference each other; owned "
+ "deletes run in declaration order, so whether this "
+ "graph purges depends on it"
+ )
+
+
+#: The phase each listener action is dispatched in by the stock callbacks:
+#: tag cleanup runs with the association deletes, permission cleanup once the
+#: entity row is gone and the captured permission name is available.
+_LISTENER_ACTION_PHASES: Mapping[ListenerAction, ExecutionPhase] = MappingProxyType(
+ {
+ ListenerAction.DELETE_TAGGED_OBJECTS: ExecutionPhase.ASSOCIATIONS,
+ ListenerAction.DELETE_DATASET_PERMISSION: ExecutionPhase.POST_DELETE,
+ }
+)
+
+
+#: Classifications the shared cleanup executes as SQL, through
+#: ``_dependency_predicates`` -- which only knows how to read an inbound
+#: foreign key.
+_EXECUTABLE_CLASSIFICATIONS: frozenset[DependencyClassification] = frozenset(
+ {
+ DependencyClassification.OWNED,
+ DependencyClassification.ASSOCIATION,
+ DependencyClassification.BLOCK,
+ }
+)
+
+
+def _validate_listener_dispatch(
+ policy: PurgeEntityPolicy, dependency: DependencyPolicy
+) -> None:
+ """Reject a listener effect no selected callback would dispatch.
+
+ Effects are dispatched from inside two stock callbacks -- association
+ cleanup and permission cleanup -- each for one phase. A declaration in any
+ other phase, or in a phase whose callback this policy replaces, is never
+ executed: the root purges and its declared cleanup silently does not run.
+ """
+ action: ListenerAction = cast(ListenerAction, dependency.listener_action)
+ expected_phase: ExecutionPhase | None = _LISTENER_ACTION_PHASES.get(action)
+ if expected_phase is None:
+ raise RuntimeError(
+ f"Unsupported listener action for {dependency.key.describe()}"
+ )
+ if dependency.phase is not expected_phase:
+ raise RuntimeError(
+ f"Listener effect {dependency.key.describe()} is declared for "
+ f"{cast(ExecutionPhase, dependency.phase).value}, but "
+ f"{action.value} is dispatched only in {expected_phase.value}"
+ )
+ if expected_phase is ExecutionPhase.ASSOCIATIONS:
+ dispatched = policy.delete_associations is delete_associations
+ callback = "delete_associations"
+ else:
+ dispatched = policy.cleanup_permission is cleanup_dataset_permission
+ callback = "cleanup_permission"
+ if not dispatched:
+ raise RuntimeError(
+ f"Listener effect {dependency.key.describe()} is dispatched by the "
+ f"stock {callback} callback, which this policy replaces"
+ )
+
+
+def _validate_dependency_declaration(
+ policy: PurgeEntityPolicy,
+ dependency: DependencyPolicy,
+ metadata: sa.MetaData,
+) -> None:
+ """Reject one declaration the shared cleanup could not carry out."""
+ key: DependencyKey = dependency.key
+ if dependency.classification in _EXECUTABLE_CLASSIFICATIONS:
+ if key.kind != "foreign_key" or key.direction != "inbound":
+ raise RuntimeError(
+ f"{dependency.classification.value} dependency "
+ f"{key.describe()} is not an inbound foreign key; the "
+ "shared cleanup can only act on those"
+ )
+ if key.related_table not in metadata.tables:
raise RuntimeError(
- f"Missing version target column for {dependency.key.describe()}"
+ f"{dependency.classification.value} dependency "
+ f"{key.describe()} names a table outside the root's "
+ "metadata, which cleanup cannot resolve"
)
+ if (
+ dependency.classification is DependencyClassification.BLOCK
+ and dependency.blocker is None
+ ):
+ raise RuntimeError(f"Missing blocker reason for {key.describe()}")
+ if dependency.classification is DependencyClassification.LISTENER_EFFECT:
+ if dependency.listener_action is None:
+ raise RuntimeError(f"Missing listener action for {key.describe()}")
+ if dependency.phase is None:
+ # Listener effects are dispatched by phase, so one without a phase
+ # matches none and would never run.
+ raise RuntimeError(f"Missing execution phase for {key.describe()}")
+ _validate_listener_dispatch(policy, dependency)
+ if (
+ dependency.classification is DependencyClassification.VERSION_OWNED
+ and dependency.version_column is None
+ ):
Review Comment:
A versioned `workflow → stage → leaf` policy can declare `leaf_version.stage_id`, but version cleanup compares that column directly with the workflow ID: purging workflow 1 with stage 10 deletes history for a surviving workflow’s stage 1 and retains stage 10’s history. Could admission reject intermediate-owner version targets until cleanup builds root-relative predicates?
##########
superset/commands/deletion_retention/purge_policy.py:
##########
@@ -860,16 +894,155 @@ def declare(
cleanup_permission=cleanup_dataset_permission,
),
}
- return validate_unique_root_policies(registry.values())
+ return tuple(registry.values())
+
+
+def _host_purge_policies(
+ provider: Callable[[], Any] | None,
+) -> tuple[PurgeEntityPolicy, ...]:
+ """Return the purge policies a host installed for its own roots.
+
+ A host distribution can carry ``SoftDeleteMixin`` entities this package
+ cannot import. The retention task discovers those entities through the
+ mixin registry, so without a policy they reach the cascade as an
+ unsupported model. The host therefore declares their purge behavior and
+ installs it under ``PURGE_POLICIES_FUNC``.
+
+ Host boundary: everything about a host policy is settled here, where it is
+ admitted -- an unavailable provider, a malformed payload, a collision with
+ a root declared in this package, two declarations for one root, and a
+ declaration the shared cleanup could not execute. Each is logged and
+ dropped, so that root is reported as unsupported instead of failing row by
+ row in a scheduled run. A broken host declaration must not take the purge
+ down with it, and must never redefine how a chart, dashboard or dataset is
+ purged.
+ """
+ if provider is None:
+ return ()
+ try:
+ provided: Any = provider()
+ except Exception: # pylint: disable=broad-except
+ logger.exception(
+ "purge_policy: %s is unavailable; keeping built-in roots only",
+ HOST_POLICIES_CONFIG_KEY,
+ )
+ return ()
+ if not isinstance(provided, (list, tuple)) or not all(
+ isinstance(policy, PurgeEntityPolicy) for policy in provided
+ ):
+ logger.error(
+ "purge_policy: %s returned %s; expected a sequence of PurgeEntityPolicy",
+ HOST_POLICIES_CONFIG_KEY,
+ type(provided).__name__,
+ )
+ return ()
+ builtin_roots: frozenset[type[Any]] = frozenset(
+ policy.model for policy in _builtin_purge_policies()
+ )
+ declared: dict[type[Any], int] = {}
+ for policy in provided:
+ declared[policy.model] = declared.get(policy.model, 0) + 1
+ # Which of two declarations for one root is authoritative is undecidable,
+ # and the loser would still delete rows. Dropping both leaves the model
+ # reported as unsupported, which is the recoverable outcome. Resolving it
+ # here also keeps the duplicate away from validate_unique_root_policies,
+ # whose ValueError would abort the whole scheduled run.
+ duplicated: list[type[Any]] = [
+ model for model, count in declared.items() if count > 1
+ ]
+ for model in duplicated:
+ logger.error(
+ "purge_policy: %s declares %d policies for %s; ignoring all of them",
+ HOST_POLICIES_CONFIG_KEY,
+ declared[model],
+ model.__name__,
+ )
+ accepted: list[PurgeEntityPolicy] = []
+ for policy in provided:
+ if policy.model in builtin_roots:
+ logger.error(
+ "purge_policy: host policy for built-in root %s ignored",
+ policy.model.__name__,
+ )
+ continue
+ if policy.model in duplicated:
+ continue
+ try:
+ _validated_policy(policy)
+ except Exception as ex: # pylint: disable=broad-except
+ # Validated at admission rather than when a row is purged: a
+ # declaration the cleanup cannot execute would otherwise fail once
+ # per eligible row, counted as a cascade failure, and stay
+ # invisible until something aged past the window.
+ logger.error(
+ "purge_policy: host policy for %s rejected: %s",
+ policy.model.__name__,
+ ex,
+ )
+ continue
+ accepted.append(policy)
+ return tuple(accepted)
-@lru_cache(maxsize=None)
-def _validated_purge_policy(model: type[Any]) -> PurgeEntityPolicy:
- """Validate and return one root policy without blocking unrelated roots."""
- try:
- policy: PurgeEntityPolicy = purge_policy_registry()[model]
- except KeyError as ex:
- raise ValueError(f"Unsupported purge model: {model.__name__}") from ex
+#: Sentinel distinguishing "no index resolved yet" from an index resolved for
+#: no provider, which is the ordinary case.
+_UNRESOLVED: Any = object()
+
+
+@dataclass
+class _ResolvedRegistry:
+ """The index built for one provider, with the roots validated so far.
+
+ Compared by provider *identity* rather than memoized with ``lru_cache``: a
+ cache keyed on the provider would hash it, and a host callable
+ implemented as a mutable dataclass instance is unhashable. That would
+ raise here -- before the host boundary could isolate the host's mistake --
+ and stop the built-in roots purging too.
+ """
+
+ provider: Any = _UNRESOLVED
+ registry: Mapping[type[Any], PurgeEntityPolicy] = field(
+ default_factory=lambda: MappingProxyType({})
+ )
+ validated: set[type[Any]] = field(default_factory=set)
+
+
+_RESOLVED: _ResolvedRegistry = _ResolvedRegistry()
+
+
+def purge_policy_registry() -> Mapping[type[Any], PurgeEntityPolicy]:
+ """Index the built-in purge roots plus any the host installed.
+
+ Rebuilt only when the installed provider changes. Freezing at first use
+ would pin whatever happened to be installed at that moment -- including
+ nothing at all, for a call made before startup finished -- until a
+ restart, while resolving on every call would invoke the provider once per
+ root and could hand back different objects each time.
+ """
+ provider: Callable[[], Any] | None = (
+ current_app.config.get(HOST_POLICIES_CONFIG_KEY) if has_app_context() else None
+ )
+ if _RESOLVED.provider is not provider:
+ host_policies: tuple[PurgeEntityPolicy, ...] = _host_purge_policies(provider)
Review Comment:
Two concurrent first-use requests can invoke the same provider: after one caches valid host policies, a slower invocation that fails transiently overwrites them with the built-in-only fallback and caches that provider identity, leaving host roots unsupported until restart or replacement. Could registry initialization be single-flight and publish its resolved state atomically?
##########
superset/commands/deletion_retention/purge_policy.py:
##########
@@ -860,16 +894,155 @@ def declare(
cleanup_permission=cleanup_dataset_permission,
),
}
- return validate_unique_root_policies(registry.values())
+ return tuple(registry.values())
+
+
+def _host_purge_policies(
+ provider: Callable[[], Any] | None,
+) -> tuple[PurgeEntityPolicy, ...]:
+ """Return the purge policies a host installed for its own roots.
+
+ A host distribution can carry ``SoftDeleteMixin`` entities this package
+ cannot import. The retention task discovers those entities through the
+ mixin registry, so without a policy they reach the cascade as an
+ unsupported model. The host therefore declares their purge behavior and
+ installs it under ``PURGE_POLICIES_FUNC``.
+
+ Host boundary: everything about a host policy is settled here, where it is
+ admitted -- an unavailable provider, a malformed payload, a collision with
+ a root declared in this package, two declarations for one root, and a
+ declaration the shared cleanup could not execute. Each is logged and
+ dropped, so that root is reported as unsupported instead of failing row by
+ row in a scheduled run. A broken host declaration must not take the purge
+ down with it, and must never redefine how a chart, dashboard or dataset is
+ purged.
+ """
+ if provider is None:
+ return ()
+ try:
+ provided: Any = provider()
+ except Exception: # pylint: disable=broad-except
+ logger.exception(
+ "purge_policy: %s is unavailable; keeping built-in roots only",
+ HOST_POLICIES_CONFIG_KEY,
+ )
+ return ()
+ if not isinstance(provided, (list, tuple)) or not all(
+ isinstance(policy, PurgeEntityPolicy) for policy in provided
+ ):
+ logger.error(
+ "purge_policy: %s returned %s; expected a sequence of PurgeEntityPolicy",
+ HOST_POLICIES_CONFIG_KEY,
+ type(provided).__name__,
+ )
+ return ()
+ builtin_roots: frozenset[type[Any]] = frozenset(
+ policy.model for policy in _builtin_purge_policies()
+ )
+ declared: dict[type[Any], int] = {}
+ for policy in provided:
+ declared[policy.model] = declared.get(policy.model, 0) + 1
+ # Which of two declarations for one root is authoritative is undecidable,
+ # and the loser would still delete rows. Dropping both leaves the model
+ # reported as unsupported, which is the recoverable outcome. Resolving it
+ # here also keeps the duplicate away from validate_unique_root_policies,
+ # whose ValueError would abort the whole scheduled run.
+ duplicated: list[type[Any]] = [
+ model for model, count in declared.items() if count > 1
+ ]
+ for model in duplicated:
+ logger.error(
+ "purge_policy: %s declares %d policies for %s; ignoring all of them",
+ HOST_POLICIES_CONFIG_KEY,
+ declared[model],
+ model.__name__,
+ )
+ accepted: list[PurgeEntityPolicy] = []
+ for policy in provided:
+ if policy.model in builtin_roots:
+ logger.error(
+ "purge_policy: host policy for built-in root %s ignored",
+ policy.model.__name__,
+ )
+ continue
+ if policy.model in duplicated:
+ continue
+ try:
+ _validated_policy(policy)
+ except Exception as ex: # pylint: disable=broad-except
+ # Validated at admission rather than when a row is purged: a
+ # declaration the cleanup cannot execute would otherwise fail once
+ # per eligible row, counted as a cascade failure, and stay
+ # invisible until something aged past the window.
+ logger.error(
+ "purge_policy: host policy for %s rejected: %s",
+ policy.model.__name__,
Review Comment:
If a host mistakenly supplies a SQLAlchemy `Table` as a policy’s `model`, validation rejects it but this logging access raises `AttributeError` outside the guard; registry initialization then fails for every built-in root too. Could malformed model values be validated or safely named inside the host-isolation boundary?
##########
superset/commands/deletion_retention/purge_policy.py:
##########
@@ -905,32 +1080,311 @@ def _validated_purge_policy(model: type[Any]) -> PurgeEntityPolicy:
details = (
f"{details}; " if details else ""
) + f"stale_listeners=[{', '.join(coverage.stale_listeners)}]"
- raise RuntimeError(f"Incomplete purge policy for {model.__name__}: {details}")
+ raise RuntimeError(
+ f"Incomplete purge policy for {policy.model.__name__}: {details}"
+ )
return policy
-def _validate_executable_declarations(policy: PurgeEntityPolicy) -> None:
- """Reject executable classifications missing their required action metadata."""
+def _ownership_dependency(
+ policy: PurgeEntityPolicy, related_table: str
+) -> DependencyPolicy | None:
+ """The single owned/association edge attaching *related_table*, if clear."""
+ candidates: tuple[DependencyPolicy, ...] = tuple(
+ dependency
+ for dependency in policy.dependencies
+ if dependency.classification
+ in {DependencyClassification.OWNED, DependencyClassification.ASSOCIATION}
+ and dependency.key.related_table == related_table
+ and dependency.key.direction == "inbound"
+ )
+ return candidates[0] if len(candidates) == 1 else None
+
+
+def _validate_owned_traversal(policy: PurgeEntityPolicy) -> None:
+ """Reject an owned table reachable only through an association.
+
+ ``cascade_hard_delete`` empties associations before owned children, while
+ an owned table's predicate selects its rows *through* its ownership path
+ (see ``_owner_value_select``). If a hop on that path is an association,
+ its rows are already gone when the owned delete runs: the statement
+ matches nothing, leaving the descendants orphaned where foreign keys are
+ unenforced and blocking the root's delete where they are not.
+
+ Refused at declaration time rather than executed. The shape has a
+ remedy -- classify the intermediate table as owned, which places it in
+ the same phase as what it leads to.
+ """
+ root_table: str = sa.inspect(policy.model).local_table.name
for dependency in policy.dependencies:
- if (
- dependency.classification is DependencyClassification.LISTENER_EFFECT
- and dependency.listener_action is None
- ):
+ if dependency.classification is not DependencyClassification.OWNED:
+ continue
+ table_name: str = dependency.key.owner_table
+ visited: set[str] = set()
+ while table_name != root_table and table_name not in visited:
+ visited.add(table_name)
+ hop: DependencyPolicy | None = _ownership_dependency(policy, table_name)
+ if hop is None:
+ # An absent or ambiguous path is reported by coverage and by
+ # _ownership_edge at execution; not this check's business.
+ break
+ if hop.classification is DependencyClassification.ASSOCIATION:
+ raise RuntimeError(
+ f"Owned dependency {dependency.key.describe()} is reachable "
+ f"only through association {hop.key.describe()}; "
+ "associations are deleted first, so the owned rows would "
+ "be orphaned"
+ )
+ table_name = hop.key.owner_table
+
+
+def _tables_share_a_foreign_key(metadata: sa.MetaData, first: str, second: str) -> bool:
+ """Whether either table declares a foreign key into the other."""
+ for owner_name, other_name in ((first, second), (second, first)):
+ table: sa.Table | None = metadata.tables.get(owner_name)
+ if table is None:
+ continue
+ for constraint in table.foreign_key_constraints:
+ if constraint.elements[0].column.table.name == other_name:
+ return True
+ return False
+
+
+def _validate_scanner_requirements(policy: PurgeEntityPolicy) -> None:
+ """Reject a root the scheduled scan cannot page through.
+
+ The retention task selects, windows and orders eligible rows by ``id``,
+ and the cascade pins each row by it. A root keyed on something else -- a
+ UUID primary key with no ``id`` column -- raises inside the scan, outside
+ the per-row error handling, so the run aborts before the remaining roots
+ are reached.
+ """
+ table: sa.Table = sa.inspect(policy.model).local_table
+ if "id" not in table.c:
Review Comment:
A host root with primary key `(tenant_id, id)` and `deleted_at` passes these checks, but the scheduled task calls `session.get(model, entity_id)` with a scalar, so every eligible row fails while dry-run reports it as purgeable. Could admission validate the mapper identity the stock task requires, or carry the complete key through scanning and lookup?
##########
tests/unit_tests/tasks/test_deletion_retention.py:
##########
@@ -330,6 +331,79 @@ class UnsupportedModel(SoftDeleteMixin):
engine.dispose()
+def test_scan_failure_keeps_the_counts_earned_before_it(app_context: None) -> None:
+ """A scan that fails part-way reports itself and keeps what it purged.
+
+ By the time a later page fails, the earlier page's deletions are
+ committed. Discarding the counts would make the run's summary and its
+ purge gauges understate what was actually removed.
+ """
+ import superset.tasks.deletion_retention as mod
+ from superset.commands.deletion_retention.purge_cascade import CascadeResult
+ from superset.models.slice import Slice
+
+ purged_result: CascadeResult = CascadeResult(
+ purged=True, entity_type="chart", entity_uuid="gone"
+ )
+
+ def pages(*args: Any, **kwargs: Any) -> Iterator[list[int]]:
+ yield [1, 2]
+ raise RuntimeError("no such column: id")
+
+ with (
+ patch.object(mod, "_iter_eligible_ids", side_effect=pages),
+ patch.object(mod, "_purge_one", return_value=purged_result),
+ ):
+ purged, would, failures, blocked, scan_failures = mod._purge_model(
+ Slice, datetime.now(), dry_run=False
+ )
+
+ assert (purged, would, failures, blocked) == (2, 0, 0, 0)
+ assert scan_failures == 1
+
+
+def test_root_without_a_table_name_does_not_abort_the_run(
+ app_config: Config,
+ monkeypatch: pytest.MonkeyPatch,
+) -> None:
+ """Resolving a root's table name is itself guarded.
+
+ The name is read off the model, so a root that cannot supply one must be
+ counted and skipped like any other failing root -- not end the pass before
+ the roots that could have purged are reached.
+ """
+ # avoid app-init regression: model helpers require the app_config fixture first.
+ from superset.models.helpers import SoftDeleteMixin
+ from superset.tasks import deletion_retention as task
+
+ supported_models: list[type[SoftDeleteMixin]] = list(task.purge_policy_registry())
+
+ class NoTableName(SoftDeleteMixin):
Review Comment:
Defining this subclass mutates the original process-global registry before `monkeypatch` saves it, so teardown restores a list still containing `NoTableName`; later retention passes unexpectedly report a scan failure for the tableless class. Could the registry be patched to a copied list before defining the subclass, as the neighboring unsupported-model test does?
##########
superset/commands/deletion_retention/purge_policy.py:
##########
@@ -905,32 +1080,311 @@ def _validated_purge_policy(model: type[Any]) -> PurgeEntityPolicy:
details = (
f"{details}; " if details else ""
) + f"stale_listeners=[{', '.join(coverage.stale_listeners)}]"
- raise RuntimeError(f"Incomplete purge policy for {model.__name__}: {details}")
+ raise RuntimeError(
+ f"Incomplete purge policy for {policy.model.__name__}: {details}"
+ )
return policy
-def _validate_executable_declarations(policy: PurgeEntityPolicy) -> None:
- """Reject executable classifications missing their required action metadata."""
+def _ownership_dependency(
+ policy: PurgeEntityPolicy, related_table: str
+) -> DependencyPolicy | None:
+ """The single owned/association edge attaching *related_table*, if clear."""
+ candidates: tuple[DependencyPolicy, ...] = tuple(
+ dependency
+ for dependency in policy.dependencies
+ if dependency.classification
+ in {DependencyClassification.OWNED, DependencyClassification.ASSOCIATION}
+ and dependency.key.related_table == related_table
+ and dependency.key.direction == "inbound"
+ )
+ return candidates[0] if len(candidates) == 1 else None
+
+
+def _validate_owned_traversal(policy: PurgeEntityPolicy) -> None:
+ """Reject an owned table reachable only through an association.
+
+ ``cascade_hard_delete`` empties associations before owned children, while
+ an owned table's predicate selects its rows *through* its ownership path
+ (see ``_owner_value_select``). If a hop on that path is an association,
+ its rows are already gone when the owned delete runs: the statement
+ matches nothing, leaving the descendants orphaned where foreign keys are
+ unenforced and blocking the root's delete where they are not.
+
+ Refused at declaration time rather than executed. The shape has a
+ remedy -- classify the intermediate table as owned, which places it in
+ the same phase as what it leads to.
+ """
+ root_table: str = sa.inspect(policy.model).local_table.name
for dependency in policy.dependencies:
- if (
- dependency.classification is DependencyClassification.LISTENER_EFFECT
- and dependency.listener_action is None
- ):
+ if dependency.classification is not DependencyClassification.OWNED:
+ continue
+ table_name: str = dependency.key.owner_table
+ visited: set[str] = set()
+ while table_name != root_table and table_name not in visited:
+ visited.add(table_name)
+ hop: DependencyPolicy | None = _ownership_dependency(policy, table_name)
+ if hop is None:
+ # An absent or ambiguous path is reported by coverage and by
+ # _ownership_edge at execution; not this check's business.
+ break
+ if hop.classification is DependencyClassification.ASSOCIATION:
+ raise RuntimeError(
+ f"Owned dependency {dependency.key.describe()} is reachable "
+ f"only through association {hop.key.describe()}; "
+ "associations are deleted first, so the owned rows would "
+ "be orphaned"
+ )
+ table_name = hop.key.owner_table
+
+
+def _tables_share_a_foreign_key(metadata: sa.MetaData, first: str, second: str) -> bool:
+ """Whether either table declares a foreign key into the other."""
+ for owner_name, other_name in ((first, second), (second, first)):
+ table: sa.Table | None = metadata.tables.get(owner_name)
+ if table is None:
+ continue
+ for constraint in table.foreign_key_constraints:
+ if constraint.elements[0].column.table.name == other_name:
+ return True
+ return False
+
+
+def _validate_scanner_requirements(policy: PurgeEntityPolicy) -> None:
+ """Reject a root the scheduled scan cannot page through.
+
+ The retention task selects, windows and orders eligible rows by ``id``,
+ and the cascade pins each row by it. A root keyed on something else -- a
+ UUID primary key with no ``id`` column -- raises inside the scan, outside
+ the per-row error handling, so the run aborts before the remaining roots
+ are reached.
+ """
+ table: sa.Table = sa.inspect(policy.model).local_table
+ if "id" not in table.c:
+ raise RuntimeError(
+ f"Purge root {policy.model.__name__} has no 'id' column; the "
+ "scheduled scan pages eligible rows by id"
+ )
+ if "deleted_at" not in table.c:
+ raise RuntimeError(
+ f"Purge root {policy.model.__name__} has no 'deleted_at' column; "
+ "every purge path requires the row to be archived first"
+ )
+
+
+def _validate_recursive_ownership(policy: PurgeEntityPolicy) -> None:
+ """Reject a self-referencing owned table under the stock cleanup.
+
+ ``delete_owned_children`` issues one statement per declared edge, so a
+ table that owns itself is pruned one level deep: a three-level tree
+ leaves its deepest rows behind -- orphaned where foreign keys are
+ unenforced, and blocking the root's own delete where they are not. A host
+ declaring this shape supplies cleanup that walks the whole sub-tree.
+ """
+ for dependency in policy.dependencies:
+ if dependency.classification is not DependencyClassification.OWNED:
+ continue
+ key: DependencyKey = dependency.key
+ if key.owner_table != key.related_table:
+ continue
+ if policy.delete_owned_children is delete_owned_children:
raise RuntimeError(
- f"Missing listener action for {dependency.key.describe()}"
+ f"Owned dependency {key.describe()} is self-referencing; the "
+ "stock owned-child cleanup deletes one level, so this policy "
+ "must supply its own delete_owned_children"
)
- if (
- dependency.classification is DependencyClassification.VERSION_OWNED
- and dependency.version_column is None
- ):
+
+
+def _validate_sibling_ownership(policy: PurgeEntityPolicy) -> None:
+ """Reject owned siblings that reference one another.
+
+ Owned deletes are sorted by ownership depth with a stable sort, so two
+ owned tables at the same depth are deleted in the order they were
+ declared. With a foreign key between them, that order decides the
+ outcome: removing the referenced table first is refused by an enforced
+ constraint and rolls the purge back. Ordering deletes by their own
+ foreign keys would be the richer fix; refusing the shape keeps the
+ engine's contract honest until something needs it.
+ """
+ metadata: sa.MetaData = sa.inspect(policy.model).local_table.metadata
+ by_depth: dict[int, list[DependencyPolicy]] = {}
+ for dependency in policy.dependencies:
+ if dependency.classification is not DependencyClassification.OWNED:
+ continue
+ depth: int = _dependency_owner_depth(policy, dependency.key)
+ by_depth.setdefault(depth, []).append(dependency)
+ for siblings in by_depth.values():
+ for index, first in enumerate(siblings):
+ for second in siblings[index + 1 :]:
+ if first.key.related_table == second.key.related_table:
+ continue
+ if _tables_share_a_foreign_key(
+ metadata, first.key.related_table, second.key.related_table
+ ):
+ raise RuntimeError(
+ f"Owned dependencies {first.key.describe()} and "
+ f"{second.key.describe()} reference each other; owned "
+ "deletes run in declaration order, so whether this "
+ "graph purges depends on it"
+ )
+
+
+#: The phase each listener action is dispatched in by the stock callbacks:
+#: tag cleanup runs with the association deletes, permission cleanup once the
+#: entity row is gone and the captured permission name is available.
+_LISTENER_ACTION_PHASES: Mapping[ListenerAction, ExecutionPhase] = MappingProxyType(
+ {
+ ListenerAction.DELETE_TAGGED_OBJECTS: ExecutionPhase.ASSOCIATIONS,
+ ListenerAction.DELETE_DATASET_PERMISSION: ExecutionPhase.POST_DELETE,
+ }
+)
+
+
+#: Classifications the shared cleanup executes as SQL, through
+#: ``_dependency_predicates`` -- which only knows how to read an inbound
+#: foreign key.
+_EXECUTABLE_CLASSIFICATIONS: frozenset[DependencyClassification] = frozenset(
+ {
+ DependencyClassification.OWNED,
+ DependencyClassification.ASSOCIATION,
+ DependencyClassification.BLOCK,
+ }
+)
+
+
+def _validate_listener_dispatch(
+ policy: PurgeEntityPolicy, dependency: DependencyPolicy
+) -> None:
+ """Reject a listener effect no selected callback would dispatch.
+
+ Effects are dispatched from inside two stock callbacks -- association
+ cleanup and permission cleanup -- each for one phase. A declaration in any
+ other phase, or in a phase whose callback this policy replaces, is never
+ executed: the root purges and its declared cleanup silently does not run.
+ """
+ action: ListenerAction = cast(ListenerAction, dependency.listener_action)
+ expected_phase: ExecutionPhase | None = _LISTENER_ACTION_PHASES.get(action)
+ if expected_phase is None:
+ raise RuntimeError(
+ f"Unsupported listener action for {dependency.key.describe()}"
+ )
+ if dependency.phase is not expected_phase:
+ raise RuntimeError(
+ f"Listener effect {dependency.key.describe()} is declared for "
+ f"{cast(ExecutionPhase, dependency.phase).value}, but "
+ f"{action.value} is dispatched only in {expected_phase.value}"
+ )
+ if expected_phase is ExecutionPhase.ASSOCIATIONS:
+ dispatched = policy.delete_associations is delete_associations
+ callback = "delete_associations"
+ else:
+ dispatched = policy.cleanup_permission is cleanup_dataset_permission
Review Comment:
A host permission listener declared at `POST_DELETE` with this stock callback passes admission, yet retaining `dataset_permission_name` for a non-dataset entity always captures `None`; the cascade skips `cleanup_permission` entirely and deletes the root without its declared permission cleanup. Could admission also check the capture prerequisite, or fail closed when a required effect has no captured name?
##########
superset/commands/deletion_retention/purge_policy.py:
##########
@@ -905,32 +1080,311 @@ def _validated_purge_policy(model: type[Any]) -> PurgeEntityPolicy:
details = (
f"{details}; " if details else ""
) + f"stale_listeners=[{', '.join(coverage.stale_listeners)}]"
- raise RuntimeError(f"Incomplete purge policy for {model.__name__}: {details}")
+ raise RuntimeError(
+ f"Incomplete purge policy for {policy.model.__name__}: {details}"
+ )
return policy
-def _validate_executable_declarations(policy: PurgeEntityPolicy) -> None:
- """Reject executable classifications missing their required action metadata."""
+def _ownership_dependency(
+ policy: PurgeEntityPolicy, related_table: str
+) -> DependencyPolicy | None:
+ """The single owned/association edge attaching *related_table*, if clear."""
+ candidates: tuple[DependencyPolicy, ...] = tuple(
+ dependency
+ for dependency in policy.dependencies
+ if dependency.classification
+ in {DependencyClassification.OWNED, DependencyClassification.ASSOCIATION}
+ and dependency.key.related_table == related_table
+ and dependency.key.direction == "inbound"
+ )
+ return candidates[0] if len(candidates) == 1 else None
+
+
+def _validate_owned_traversal(policy: PurgeEntityPolicy) -> None:
+ """Reject an owned table reachable only through an association.
+
+ ``cascade_hard_delete`` empties associations before owned children, while
+ an owned table's predicate selects its rows *through* its ownership path
+ (see ``_owner_value_select``). If a hop on that path is an association,
+ its rows are already gone when the owned delete runs: the statement
+ matches nothing, leaving the descendants orphaned where foreign keys are
+ unenforced and blocking the root's delete where they are not.
+
+ Refused at declaration time rather than executed. The shape has a
+ remedy -- classify the intermediate table as owned, which places it in
+ the same phase as what it leads to.
+ """
+ root_table: str = sa.inspect(policy.model).local_table.name
for dependency in policy.dependencies:
- if (
- dependency.classification is DependencyClassification.LISTENER_EFFECT
- and dependency.listener_action is None
- ):
+ if dependency.classification is not DependencyClassification.OWNED:
+ continue
+ table_name: str = dependency.key.owner_table
+ visited: set[str] = set()
+ while table_name != root_table and table_name not in visited:
+ visited.add(table_name)
+ hop: DependencyPolicy | None = _ownership_dependency(policy, table_name)
+ if hop is None:
+ # An absent or ambiguous path is reported by coverage and by
+ # _ownership_edge at execution; not this check's business.
+ break
+ if hop.classification is DependencyClassification.ASSOCIATION:
+ raise RuntimeError(
+ f"Owned dependency {dependency.key.describe()} is reachable "
+ f"only through association {hop.key.describe()}; "
+ "associations are deleted first, so the owned rows would "
+ "be orphaned"
+ )
+ table_name = hop.key.owner_table
+
+
+def _tables_share_a_foreign_key(metadata: sa.MetaData, first: str, second: str) -> bool:
+ """Whether either table declares a foreign key into the other."""
+ for owner_name, other_name in ((first, second), (second, first)):
+ table: sa.Table | None = metadata.tables.get(owner_name)
+ if table is None:
+ continue
+ for constraint in table.foreign_key_constraints:
+ if constraint.elements[0].column.table.name == other_name:
+ return True
+ return False
+
+
+def _validate_scanner_requirements(policy: PurgeEntityPolicy) -> None:
+ """Reject a root the scheduled scan cannot page through.
+
+ The retention task selects, windows and orders eligible rows by ``id``,
+ and the cascade pins each row by it. A root keyed on something else -- a
+ UUID primary key with no ``id`` column -- raises inside the scan, outside
+ the per-row error handling, so the run aborts before the remaining roots
+ are reached.
+ """
+ table: sa.Table = sa.inspect(policy.model).local_table
+ if "id" not in table.c:
+ raise RuntimeError(
+ f"Purge root {policy.model.__name__} has no 'id' column; the "
+ "scheduled scan pages eligible rows by id"
+ )
+ if "deleted_at" not in table.c:
+ raise RuntimeError(
+ f"Purge root {policy.model.__name__} has no 'deleted_at' column; "
+ "every purge path requires the row to be archived first"
+ )
+
+
+def _validate_recursive_ownership(policy: PurgeEntityPolicy) -> None:
+ """Reject a self-referencing owned table under the stock cleanup.
+
+ ``delete_owned_children`` issues one statement per declared edge, so a
+ table that owns itself is pruned one level deep: a three-level tree
+ leaves its deepest rows behind -- orphaned where foreign keys are
+ unenforced, and blocking the root's own delete where they are not. A host
+ declaring this shape supplies cleanup that walks the whole sub-tree.
+ """
+ for dependency in policy.dependencies:
+ if dependency.classification is not DependencyClassification.OWNED:
+ continue
+ key: DependencyKey = dependency.key
+ if key.owner_table != key.related_table:
+ continue
+ if policy.delete_owned_children is delete_owned_children:
raise RuntimeError(
- f"Missing listener action for {dependency.key.describe()}"
+ f"Owned dependency {key.describe()} is self-referencing; the "
+ "stock owned-child cleanup deletes one level, so this policy "
+ "must supply its own delete_owned_children"
)
- if (
- dependency.classification is DependencyClassification.VERSION_OWNED
- and dependency.version_column is None
- ):
+
+
+def _validate_sibling_ownership(policy: PurgeEntityPolicy) -> None:
+ """Reject owned siblings that reference one another.
+
+ Owned deletes are sorted by ownership depth with a stable sort, so two
+ owned tables at the same depth are deleted in the order they were
+ declared. With a foreign key between them, that order decides the
+ outcome: removing the referenced table first is refused by an enforced
+ constraint and rolls the purge back. Ordering deletes by their own
+ foreign keys would be the richer fix; refusing the shape keeps the
+ engine's contract honest until something needs it.
+ """
+ metadata: sa.MetaData = sa.inspect(policy.model).local_table.metadata
+ by_depth: dict[int, list[DependencyPolicy]] = {}
+ for dependency in policy.dependencies:
+ if dependency.classification is not DependencyClassification.OWNED:
+ continue
+ depth: int = _dependency_owner_depth(policy, dependency.key)
+ by_depth.setdefault(depth, []).append(dependency)
+ for siblings in by_depth.values():
+ for index, first in enumerate(siblings):
+ for second in siblings[index + 1 :]:
+ if first.key.related_table == second.key.related_table:
+ continue
+ if _tables_share_a_foreign_key(
+ metadata, first.key.related_table, second.key.related_table
+ ):
+ raise RuntimeError(
+ f"Owned dependencies {first.key.describe()} and "
+ f"{second.key.describe()} reference each other; owned "
+ "deletes run in declaration order, so whether this "
+ "graph purges depends on it"
+ )
+
+
+#: The phase each listener action is dispatched in by the stock callbacks:
+#: tag cleanup runs with the association deletes, permission cleanup once the
+#: entity row is gone and the captured permission name is available.
+_LISTENER_ACTION_PHASES: Mapping[ListenerAction, ExecutionPhase] = MappingProxyType(
+ {
+ ListenerAction.DELETE_TAGGED_OBJECTS: ExecutionPhase.ASSOCIATIONS,
+ ListenerAction.DELETE_DATASET_PERMISSION: ExecutionPhase.POST_DELETE,
+ }
+)
+
+
+#: Classifications the shared cleanup executes as SQL, through
+#: ``_dependency_predicates`` -- which only knows how to read an inbound
+#: foreign key.
+_EXECUTABLE_CLASSIFICATIONS: frozenset[DependencyClassification] = frozenset(
+ {
+ DependencyClassification.OWNED,
+ DependencyClassification.ASSOCIATION,
+ DependencyClassification.BLOCK,
+ }
+)
+
+
+def _validate_listener_dispatch(
+ policy: PurgeEntityPolicy, dependency: DependencyPolicy
+) -> None:
+ """Reject a listener effect no selected callback would dispatch.
+
+ Effects are dispatched from inside two stock callbacks -- association
+ cleanup and permission cleanup -- each for one phase. A declaration in any
+ other phase, or in a phase whose callback this policy replaces, is never
+ executed: the root purges and its declared cleanup silently does not run.
+ """
+ action: ListenerAction = cast(ListenerAction, dependency.listener_action)
+ expected_phase: ExecutionPhase | None = _LISTENER_ACTION_PHASES.get(action)
+ if expected_phase is None:
+ raise RuntimeError(
+ f"Unsupported listener action for {dependency.key.describe()}"
+ )
+ if dependency.phase is not expected_phase:
+ raise RuntimeError(
+ f"Listener effect {dependency.key.describe()} is declared for "
+ f"{cast(ExecutionPhase, dependency.phase).value}, but "
+ f"{action.value} is dispatched only in {expected_phase.value}"
+ )
+ if expected_phase is ExecutionPhase.ASSOCIATIONS:
+ dispatched = policy.delete_associations is delete_associations
+ callback = "delete_associations"
+ else:
+ dispatched = policy.cleanup_permission is cleanup_dataset_permission
+ callback = "cleanup_permission"
+ if not dispatched:
+ raise RuntimeError(
+ f"Listener effect {dependency.key.describe()} is dispatched by the "
+ f"stock {callback} callback, which this policy replaces"
+ )
+
+
+def _validate_dependency_declaration(
+ policy: PurgeEntityPolicy,
+ dependency: DependencyPolicy,
+ metadata: sa.MetaData,
+) -> None:
+ """Reject one declaration the shared cleanup could not carry out."""
+ key: DependencyKey = dependency.key
+ if dependency.classification in _EXECUTABLE_CLASSIFICATIONS:
+ if key.kind != "foreign_key" or key.direction != "inbound":
+ raise RuntimeError(
+ f"{dependency.classification.value} dependency "
+ f"{key.describe()} is not an inbound foreign key; the "
+ "shared cleanup can only act on those"
+ )
+ if key.related_table not in metadata.tables:
raise RuntimeError(
- f"Missing version target column for {dependency.key.describe()}"
+ f"{dependency.classification.value} dependency "
+ f"{key.describe()} names a table outside the root's "
Review Comment:
With both default-schema `root`/`child` and `host.root`/`host.child` in the metadata, a policy for `host.root` passes admission but stock cleanup resolves the unqualified names to the default-schema tables; purging host root 1 can delete children of unrelated live default-schema roots. Could schema-qualified table identities be preserved throughout discovery and execution, or this shape be rejected before admission?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(retention): let a host install purge policies for its own soft-delete roots [superset]
Posted by "bito-code-review[bot] (via GitHub)" <gi...@apache.org>.
bito-code-review[bot] commented on PR #44892:
URL: https://github.com/apache/superset/pull/44892#issuecomment-5951144830
AI Code Review is in progress (usually takes 3 to 15 minutes unless it's a very large PR).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(retention): let a host install purge policies for its own soft-delete roots [superset]
Posted by "codecov[bot] (via GitHub)" <gi...@apache.org>.
codecov[bot] commented on PR #44892:
URL: https://github.com/apache/superset/pull/44892#issuecomment-5951457657
## [Codecov](https://app.codecov.io/gh/apache/superset/pull/44892?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:x: Patch coverage is `34.48276%` with `19 lines` in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 65.90%. Comparing base ([`c43770e`](https://app.codecov.io/gh/apache/superset/commit/c43770e12da9a812d9cd2acba759eb99c68aa107?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`ee7adb9`](https://app.codecov.io/gh/apache/superset/commit/ee7adb9414d9cbf9cde8e208b01d95f654b48175?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
:warning: Report is 3 commits behind head on master.
| [Files with missing lines](https://app.codecov.io/gh/apache/superset/pull/44892?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [...perset/commands/deletion\_retention/purge\_policy.py](https://app.codecov.io/gh/apache/superset/pull/44892?src=pr&el=tree&filepath=superset%2Fcommands%2Fdeletion_retention%2Fpurge_policy.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-c3VwZXJzZXQvY29tbWFuZHMvZGVsZXRpb25fcmV0ZW50aW9uL3B1cmdlX3BvbGljeS5weQ==) | 32.14% | [17 Missing and 2 partials :warning: ](https://app.codecov.io/gh/apache/superset/pull/44892?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #44892 +/- ##
===========================================
- Coverage 82.05% 65.90% -16.16%
===========================================
Files 2991 2991
Lines 184538 184565 +27
Branches 42722 42727 +5
===========================================
- Hits 151431 121632 -29799
- Misses 30355 60259 +29904
+ Partials 2752 2674 -78
```
| [Flag](https://app.codecov.io/gh/apache/superset/pull/44892/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [hive](https://app.codecov.io/gh/apache/superset/pull/44892/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `36.34% <20.68%> (-0.01%)` | :arrow_down: |
| [mysql](https://app.codecov.io/gh/apache/superset/pull/44892/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [presto](https://app.codecov.io/gh/apache/superset/pull/44892/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `38.22% <20.68%> (-0.01%)` | :arrow_down: |
| [python](https://app.codecov.io/gh/apache/superset/pull/44892/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `55.02% <34.48%> (-31.05%)` | :arrow_down: |
| [sqlite](https://app.codecov.io/gh/apache/superset/pull/44892/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `54.89% <34.48%> (-0.01%)` | :arrow_down: |
| [unit](https://app.codecov.io/gh/apache/superset/pull/44892/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
</details>
[:umbrella: View full report in Codecov by Harness](https://app.codecov.io/gh/apache/superset/pull/44892?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
- :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(retention): let a host install purge policies for its own soft-delete roots [superset]
Posted by "alexandrusoare (via GitHub)" <gi...@apache.org>.
alexandrusoare commented on code in PR #44892:
URL: https://github.com/apache/superset/pull/44892#discussion_r4165609720
##########
docs/docs/using-superset/recently-archived.mdx:
##########
@@ -123,7 +123,18 @@ Each skip increments `deletion_retention.unsupported_models.<table>` once per
run. Unsupported models are excluded from `would_purge` and do not count as
`cascade_failures`; supported models continue to use their existing policies.
An empty `unsupported_models` map means no registered models were skipped for
-this reason. This reporting does not add purge support for host-defined models.
+this reason. This reporting does not itself add purge support for host-defined
+models.
+
+A host distribution that carries its own soft-deletable entities can supply
+their purge behavior by installing `PURGE_POLICIES_FUNC`: a callable returning
+one `PurgeEntityPolicy` per root it owns. Installed policies are validated
+against the discovered dependency graph exactly as the built-in ones are, and a
+model that gains an installed policy stops being reported as unsupported. A
+provider that fails, returns something other than a sequence of policies, or
+redeclares a built-in chart, dashboard or dataset root is logged and ignored,
+leaving the built-in roots unchanged. Hosts can require
+`superset.versioning.utils.HOST_POLICY_API_VERSION` before installing.
Review Comment:
Agreed — I'd already dropped the version bump, so that line pointed at a signal that no longer exists. The docs now say to check for HOST_POLICIES_CONFIG_KEY in the purge module, which tracks the hook directly and fails closed on a rollback.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org