You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@superset.apache.org by "rusackas (via GitHub)" <gi...@apache.org> on 2026/09/17 22:46:48 UTC
[PR] fix(charts): decouple annotation cache from the shared dataframe cache [superset]
rusackas opened a new pull request, #44406:
URL: https://github.com/apache/superset/pull/44406
### SUMMARY
A prior fix (#42930) closed a real cross-user data leak: annotation-layer
payloads are fetched per requesting user and, for chart-backed layers, scoped
by the RLS clauses of the referenced chart's datasource, so they need to be
isolated per viewer. That fix bound the requesting user's identity (and RLS
context) into the cache key for the *entire* chart dataframe, since the
dataframe and the annotation payload shared one cache entry. The side effect:
every distinct viewer of an annotation-layer chart now gets their own full
copy of the (potentially much larger) dataframe cached separately, instead of
sharing one entry.
This PR splits the two apart. The dataframe's cache key goes back to
depending only on its own datasource/RLS/`extra_cache_keys` (unscoped by
requesting user), so distinct viewers of the same chart share one dataframe
cache entry again. Annotation data is now resolved and cached under its own,
separate key that still carries the per-user/per-RLS security context #42930
introduced β so the original leak stays fixed, but only the (much smaller)
annotation payload pays the per-user cost.
As a side effect, this also restores cross-user task dedup for annotated
charts in the async ("get the fresh data") flow: the SHARED task key there
is derived from the same dataframe cache key, so it had the identical
coupling problem β two users viewing the same annotated chart could no
longer join the same in-flight query task.
No cache migration is needed; keys are content-addressed and recomputed on
every request, so stale entries under the old key shape simply age out via
normal TTL.
### TESTING INSTRUCTIONS
- `pytest tests/unit_tests/common/test_query_context_processor.py` β includes
new coverage: the dataframe cache key no longer varies per user for
annotated queries, the annotation cache key still does, `_get_annotation_data_cached`'s
hit/miss/`force_query`/`force_cached` behavior, and an end-to-end case
proving a dataframe cache hit skips recomputation while annotation data
still resolves through its own path into the payload.
- `pytest tests/unit_tests/tasks/test_async_queries.py` β unaffected/still green.
- Manually: load a dashboard with an annotation-layer chart as two different
users with different RLS; confirm both see correctly-scoped annotation data
and that the chart's underlying dataframe query only runs once (e.g. via
query count/logging) rather than once per viewer.
### ADDITIONAL INFORMATION
- [ ] Has associated issue:
- [ ] Required feature flags:
- [ ] Changes UI
- [ ] Includes DB Migration (follow approval process in [SIP-59](https://github.com/apache/superset/issues/13351))
- [ ] Migration is atomic, supports rollback & is backwards-compatible
- [ ] Confirm DB migration upgrade and downgrade tested
- [ ] Runtime estimates and downtime expectations provided
- [ ] Introduces new feature or API
- [ ] Removes existing feature or API
π€ Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] fix(charts): decouple annotation cache from the shared dataframe cache [superset]
Posted by "codeant-ai-for-open-source[bot] (via GitHub)" <gi...@apache.org>.
codeant-ai-for-open-source[bot] commented on PR #44406:
URL: https://github.com/apache/superset/pull/44406#issuecomment-6056192656
## CodeAnt PR Risk: Low Risk
- The PR appears safe to merge based on the supplied code and review conversations.
- Annotation payloads use a separate access-scoped cache, while dataframe results remain shareable across viewers.
- Tests cover access-scope separation, unresolved scopes, cache misses, and independent forced refresh behavior.
**Assessed commit:** `4e7dee066cd6`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] fix(charts): decouple annotation cache from the shared dataframe cache [superset]
Posted by "netlify[bot] (via GitHub)" <gi...@apache.org>.
netlify[bot] commented on PR #44406:
URL: https://github.com/apache/superset/pull/44406#issuecomment-5727891032
### <span aria-hidden="true">β
</span> Deploy Preview for *superset-docs-preview* ready!
| Name | Link |
|:-:|------------------------|
|<span aria-hidden="true">π¨</span> Latest commit | fdc56d4a6400cd9ca76ea929fc7a93c571b4af92 |
|<span aria-hidden="true">π</span> Latest deploy log | https://app.netlify.com/projects/superset-docs-preview/deploys/6aad00c1237a2f0009f71b98 |
|<span aria-hidden="true">π</span> Deploy Preview | [https://deploy-preview-44406--superset-docs-preview.netlify.app](https://deploy-preview-44406--superset-docs-preview.netlify.app) |
|<span aria-hidden="true">π±</span> Preview on mobile | <details><summary> Toggle QR Code... </summary><br /><br /><br /><br />_Use your smartphone camera to open QR code link._</details> |
|<span aria-hidden="true">π€</span> Make changes | [Run an agent on this branch](https://app.netlify.com/projects/superset-docs-preview/agent-runs#DcoxDoAgDADAv3TWWDUR9Qc-oxQIJNoSZdAY_y7bDfdCPvXIBVaABuxJwrE6pLsjES1UkkrLxNG3F2v2dTmfd302V99E5BC5H0ZDQ0DEJZjeLjN8Pw) |
---
<!-- [superset-docs-preview Preview](https://deploy-preview-44406--superset-docs-preview.netlify.app) -->
_To edit notification comments on pull requests, go to your [Netlify project configuration](https://app.netlify.com/projects/superset-docs-preview/configuration/notifications#deploy-notifications)._
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] fix(charts): decouple annotation cache from the shared dataframe cache [superset]
Posted by "bito-code-review[bot] (via GitHub)" <gi...@apache.org>.
bito-code-review[bot] commented on PR #44406:
URL: https://github.com/apache/superset/pull/44406#issuecomment-5902353538
AI Code Review is in progress (usually takes 3 to 15 minutes unless it's a very large PR).
<details>
<summary>
Bito Usage Guide
</summary>
**Commands**
Type the following command in the pull request comment and save the comment.
- `/review` - Manually triggers an incremental AI Review.
- `/review full` - Manually triggers a full AI Review.
- `/pause` - Pauses automatic reviews on this pull request.
- `/resume` - Resumes automatic reviews.
- `/resolve` - Marks all Bito-posted review comments as resolved.
- `/abort` - Cancels all in-progress reviews.
Refer to the <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/available-commands" target="_blank" rel="noopener noreferrer">documentation</a> for additional commands.
**Configuration**
This repository uses `Superset` You can customize the agent settings <a href="https://alpha.bito.ai/home/ai-agents/code-review-agent" target="_blank" rel="noopener noreferrer">here</a> or contact your Bito workspace admin at evan@preset.io.
**Documentation & Help**
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/getting-started/install-run-using-bito-cloud/create-or-customize-an-agent-instance" target="_blank" rel="noopener noreferrer">Customize agent settings</a>
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/implementing-custom-code-review-rules" target="_blank" rel="noopener noreferrer">Review rules</a>
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent" target="_blank" rel="noopener noreferrer">General documentation</a>
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/faqs" target="_blank" rel="noopener noreferrer">FAQ</a>
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] fix(charts): decouple annotation cache from the shared dataframe cache [superset]
Posted by "sadpandajoe (via GitHub)" <gi...@apache.org>.
sadpandajoe commented on code in PR #44406:
URL: https://github.com/apache/superset/pull/44406#discussion_r4264573946
##########
tests/unit_tests/common/test_query_context_processor.py:
##########
@@ -126,25 +131,350 @@ def processor(mock_query_context):
return processor
-def test_query_cache_key_binds_annotation_data_to_requesting_user(processor):
- """The cache key for annotated queries must differ per requesting user."""
+def test_annotation_cache_key_binds_native_annotation_read_scope(processor) -> None:
+ """The annotation cache key for NATIVE layers must differ when the
+ requester's ``can_read`` (Annotation) access differs -- not who they are."""
query_obj = MagicMock()
query_obj.annotation_layers = [{"sourceType": "NATIVE", "name": "a", "value": 1}]
- with (
- patch(
- "superset.common.query_context_processor.get_user_id",
- side_effect=[1, 2],
- ),
- patch("superset.common.query_context_processor.security_manager"),
- ):
- processor.query_cache_key(query_obj)
- processor.query_cache_key(query_obj)
+ # ``security_manager`` autodetects as an async spec under a bare
+ # ``patch()`` (its real object trips ``unittest.mock``'s coroutine
+ # inference), which would silently turn every attribute access into an
+ # ``AsyncMock`` returning a fresh unawaited coroutine per call -- always
+ # unequal to itself and never equal to a configured return value. Forcing
+ # ``new_callable=MagicMock`` keeps these synchronous, as the real object
+ # is.
+ with patch(
+ "superset.common.query_context_processor.security_manager",
+ new_callable=MagicMock,
+ ) as security_manager:
+ security_manager.can_access.side_effect = [True, False]
+ processor.annotation_cache_key(query_obj)
+ processor.annotation_cache_key(query_obj)
contexts = [
call.kwargs["annotation_context"] for call in query_obj.cache_key.call_args_list
]
assert contexts[0] != contexts[1]
+def test_annotation_cache_key_shares_across_same_access_scope() -> None:
+ """Two distinct requesters (separate processor/query-object instances,
+ standing in for two different requests) with identical access scope must
+ produce identical annotation-context material. Reusing a single
+ processor/query_obj across both calls (as this test previously did)
+ would pass trivially regardless of whether the key is scope-based or
+ identity-based, since nothing about "who's asking" would ever vary."""
+ layer = {"sourceType": "NATIVE", "name": "a", "value": 1}
+ processor_a = QueryContextProcessor(MagicMock())
+ processor_b = QueryContextProcessor(MagicMock())
+ query_obj_a = MagicMock(annotation_layers=[layer])
+ query_obj_b = MagicMock(annotation_layers=[layer])
+
+ with patch(
+ "superset.common.query_context_processor.security_manager",
+ new_callable=MagicMock,
+ ) as security_manager:
+ security_manager.can_access.return_value = True
+ context_a = processor_a._annotation_cache_context(query_obj_a)
+ context_b = processor_b._annotation_cache_context(query_obj_b)
+
+ assert context_a == context_b
Review Comment:
This test never varies the requesting user: both processors run under the same ambient identity, so putting `get_user_id()` back into the annotation context would still give equal contexts and pass. That is the exact regression this PR is meant to prevent (same-scope viewers sharing one entry).
Could it patch `get_user_id` to return different values for the two calls and assert the contexts stay equal?
##########
superset/common/query_context_processor.py:
##########
@@ -470,72 +499,223 @@ def query_cache_key(self, query_obj: QueryObject, **kwargs: Any) -> str | None:
)
return cache_key
- def _annotation_cache_context(self, query_obj: QueryObject) -> dict[str, Any]:
+ def annotation_cache_key(self, query_obj: QueryObject) -> str | None:
"""
- Cache-key material binding cached annotation data to its security
- context.
-
- Annotation payloads are fetched per requesting user and stored on the
- same cache entry as the dataframe, so the key also binds the requesting
- user and, for chart-backed layers, the RLS clauses of the referenced
- chart's datasource.
+ Cache key for this query's annotation-layer payload, or ``None`` when
+ the query has no annotation layers or when some layer's access scope
+ could not be resolved (the payload is then never cached, see
+ :meth:`_annotation_source_scope`).
+
+ Annotation payloads are fetched under the requesting user's access
+ scope, which is a stricter security requirement than the dataframe
+ itself has. Keying them separately from :meth:`query_cache_key` keeps
+ that scoping from forcing every distinct viewer of an annotated chart
+ onto their own full copy of the (potentially much larger) shared
+ dataframe: users with the same access scope share this key too.
"""
- source_rls: dict[str, list[str] | None] = {}
- source_versions: dict[str, tuple[str, str]] = {}
- metadata_versions: dict[str, str] = {}
+ if not query_obj or not query_obj.annotation_layers:
+ return None
+ if (annotation_context := self._annotation_cache_context(query_obj)) is None:
+ return None
+ return self.query_cache_key(query_obj, annotation_context=annotation_context)
+
+ def _annotation_cache_context(
+ self, query_obj: QueryObject
+ ) -> dict[str, Any] | None:
+ """
+ Cache-key material binding annotation data to its security *scope* so
+ users with the same access share a cache entry and users with a
+ different scope β or no access β never read each other's data.
+
+ * NATIVE layers: the ``can_read`` permission on ``Annotation``, the
+ only user-dependent dimension of these global records.
+ * Chart-backed (``line``/``table``) layers: see
+ :meth:`_annotation_source_scope`, recorded per *layer* (not per
+ referenced chart) since two layers can reference the same chart
+ with different overrides, and so different scopes.
+
+ Returns ``None`` when any layer's scope cannot be resolved, meaning
+ the payload must not be cached at all.
+ """
+ context: dict[str, Any] = {}
+
+ if any(
+ layer.get("sourceType") == "NATIVE" for layer in query_obj.annotation_layers
+ ):
+ context["annotation_read"] = security_manager.can_access(
+ "can_read", "Annotation"
+ )
+
+ source_scope: list[dict[str, Any]] = []
for layer in query_obj.annotation_layers:
if (
layer.get("sourceType")
not in ANNOTATION_SOURCE_TYPES_WITH_CHART_REFERENCE
):
continue
- layer_value = layer.get("value")
+ if (scope := self._annotation_source_scope(layer)) is None:
+ return None
+ source_scope.append({"value": str(layer.get("value")), **scope})
+ if source_scope:
+ context["source_scope"] = source_scope
+
+ return context
+
+ def _annotation_source_scope(self, layer: dict[str, Any]) -> dict[str, Any] | None:
+ """
+ Access and data-identity cache-key material for one chart-backed
+ annotation layer, or ``None`` when it cannot be resolved.
+
+ ``None`` means "do not cache": a missing chart, a chart with neither
+ a saved query context nor a resolvable datasource, or any error while
+ deriving the scope would otherwise collapse an authorized and a
+ denied requester onto one shared identity, letting the latter read
+ the former's warmed payload. The caller fetches live instead.
+
+ ``access`` keeps a user denied the referenced chart's datasource from
+ reading an authorized user's cached payload. When the chart has a
+ saved query context, this runs the *same* authorization path
+ :meth:`get_viz_annotation_data` executes
+ (``QueryContext.raise_for_access``) rather than the coarser
+ :meth:`SecurityManager.can_access_datasource` β a requester whose
+ access comes from a dashboard/viewer-promiscuous-mode bypass (which
+ depends on the chart's own saved ``form_data``, e.g. its
+ ``slice_id``/``dashboardId``) would otherwise still fail that coarser,
+ context-free check and collapse onto the same denied scope as a
+ genuinely unauthorized requester, letting the latter read the
+ former's cached payload. ``data_key`` is the annotation chart's own
+ query cache key(s) β derived from the same, override-applied query
+ objects actually executed (see :meth:`_apply_annotation_overrides`),
+ so it captures the datasource version, RLS clauses, and any per-user
+ Jinja/virtual-dataset RLS material exactly as the live fetch would,
+ including material an override only introduces at a finer grain.
+ Reusing this logic avoids re-deriving it and automatically inherits
+ any future correctness fixes made there.
+ """
+ layer_value = layer.get("value")
+ try:
chart = (
ChartDAO.find_by_id(layer_value) if layer_value is not None else None
)
- annotation_datasource: Datasource | None = (
- chart.resolved_datasource if chart else None
- )
- if chart and isinstance(chart.query_context, str):
+ if chart is None:
+ return None
+
+ # The saved query context is what the live fetch executes and
+ # authorizes against, so it is consulted first, independent of
+ # the chart's own datasource pointer: a chart can have its
+ # ``datasource_id`` cleared while its saved query context still
+ # targets a valid dataset, and the live fetch would still work.
+ annotation_query_context = chart.get_query_context()
+ if annotation_query_context is not None:
+ self._apply_annotation_overrides(annotation_query_context, layer)
try:
- source: dict[str, Any] = json.loads(chart.query_context)[
- "datasource"
- ]
- annotation_datasource = DatasourceDAO.get_datasource(
- source["type"], source["id"]
- )
- except (
- ValueError,
- TypeError,
- KeyError,
- DatasourceNotFound,
- DatasourceTypeNotSupportedError,
- DatasourceValueIsIncorrect,
- ):
- annotation_datasource = None
+ annotation_query_context.raise_for_access()
+ access: Any = True
+ except SupersetSecurityException:
+ access = False
+ data_key: Any = [
+ annotation_query_context.query_cache_key(query_object)
+ for query_object in annotation_query_context.queries
+ ]
+ annotation_datasource: Any = annotation_query_context.datasource
+ else:
+ # Fall back to the RLS-clause identity when the chart has no
+ # saved query context to key on. resolved_datasource, not
+ # datasource: the latter is pinned to table-backed
+ # datasources and resolves to None for a semantic-view-backed
+ # chart.
+ datasource = chart.resolved_datasource
+ if datasource is None:
+ return None
+ access = security_manager.can_access_datasource(datasource)
+ data_key = security_manager.get_rls_cache_key(datasource)
+ annotation_datasource = datasource
+ scope: dict[str, Any] = {"access": access, "data_key": data_key}
+ # Semantic-view results are versioned by the layer's metadata
+ # generation and result discriminator, which the datasource-level
+ # key material above does not carry.
if isinstance(annotation_datasource, SemanticView):
- metadata_versions[str(layer_value)] = (
- annotation_datasource.metadata_generation
- )
- source_rls[str(layer.get("value"))] = (
- security_manager.get_rls_cache_key(annotation_datasource)
- if annotation_datasource
- else None
+ scope["metadata_version"] = annotation_datasource.metadata_generation
+ if (
+ discriminator := annotation_datasource.result_cache_discriminator
+ ) is not None:
+ scope["result_version"] = discriminator
+ except Exception: # noqa: BLE001 pylint: disable=broad-except
+ # Derivation can fail well beyond SupersetException: the DAO
+ # lookup and lazy ``datasource`` load are themselves live DB
+ # queries, and the RLS lookup / a virtual dataset's
+ # get_extra_cache_keys() renders Jinja, either of which can raise
+ # a driver/template error. None of that should ever 500 the whole
+ # chart-data request; fail closed instead. A fallback scope
+ # shared by every requester whose derivation failed would still
+ # be a shared identity, so the answer is "no caching", not a
+ # substitute key.
+ logger.warning(
+ "Could not derive annotation cache key for chart %s; "
+ "annotation data for this request will not be cached",
+ layer_value,
+ exc_info=True,
)
- if isinstance(annotation_datasource, SemanticView):
- discriminator: tuple[str, str] | None = (
- annotation_datasource.result_cache_discriminator
+ return None
+ return scope
+
+ def _get_annotation_data_cached(
+ self,
+ query_obj: QueryObject,
+ cache_key: str | None,
+ force_query: bool,
+ force_cached: bool | None,
+ timeout: int | None,
+ datasource_uid: str | None,
+ ) -> dict[str, Any]:
+ """
+ Fetch this query's annotation-layer payload, cached under its own
+ (user/RLS-scoped) entry, separate from the shared dataframe cache.
+
+ A ``None`` ``cache_key`` means some layer's access scope could not be
+ resolved (see :meth:`_annotation_source_scope`), so there is no key
+ under which a cached payload could be safely shared: an unresolved
+ scope looks the same for an authorized and a denied requester. The
+ payload is then fetched live and never cached. That holds under
+ ``force_cached`` too: computing here beats failing the read-back of
+ an otherwise successful async task.
+ """
+ if cache_key is None:
+ return self.get_annotation_data(query_obj)
Review Comment:
When an annotation layer's scope can't be resolved, this live fetch runs even under `force_cached`. In the async probe (`_run_async` only catches the cache-load error), a failed fetch now surfaces as an uncaught `ChartDataQueryFailedError` instead of falling through to the task path.
Concretely: user A warms the shared dataframe for a chart whose annotation layer references a chart that user B can't see. B loads it in async mode, hits the warm dataframe, the live fetch fails with "chart not found", and B gets a server error where the old code (cache miss, then task) and the sync path return a 400.
Should the `force_cached` probe return a cache-load miss for an unresolved scope instead of computing, so the request still goes through the normal task path?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] fix(charts): decouple annotation cache from the shared dataframe cache [superset]
Posted by "rusackas (via GitHub)" <gi...@apache.org>.
rusackas commented on code in PR #44406:
URL: https://github.com/apache/superset/pull/44406#discussion_r4297990070
##########
superset/common/query_context_processor.py:
##########
@@ -354,6 +359,30 @@ def get_df_payload_result(
# nonce reads the freshly-cached result instead of recomputing it.
self._mark_force_executed(query_obj, cache_key, cache.result_persisted)
+ # Annotation data is fetched per requesting user (and, for chart-backed
+ # layers, scoped by the referenced chart datasource's RLS), so it is
+ # resolved and cached under its own entry β independent of whether the
+ # (shareable) dataframe above was a hit or a miss β rather than forcing
+ # every viewer of the same chart onto their own full dataframe copy.
+ annotation_data: dict[str, Any] = {}
+ if (
+ query_obj
+ and query_obj.annotation_layers
+ and cache.status != QueryStatus.FAILED
+ ):
+ try:
+ annotation_data = self._get_annotation_data_cached(
+ query_obj=query_obj,
+ cache_key=annotation_key,
+ force_query=annotation_force_query,
+ force_cached=force_cached,
+ timeout=self.get_cache_timeout(),
+ datasource_uid=self._qc_datasource.uid,
+ )
+ except QueryObjectValidationError as ex:
Review Comment:
Good catch. The annotation catch now re-raises `SemanticResultCompletenessError` before the generic `QueryObjectValidationError` handler, same as the dataframe path. Added a `get_df_payload_result` test asserting it escapes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] fix(charts): decouple annotation cache from the shared dataframe cache [superset]
Posted by "sadpandajoe (via GitHub)" <gi...@apache.org>.
sadpandajoe commented on code in PR #44406:
URL: https://github.com/apache/superset/pull/44406#discussion_r4321202387
##########
tests/unit_tests/tasks/test_query_cancel.py:
##########
@@ -159,6 +159,26 @@ def test_capture_registers_abort_handler_once_when_id_available() -> None:
assert ordered.index("set_cancellation") < ordered.index("on_abort")
+def test_capture_ignores_cursor_from_a_different_database() -> None:
Review Comment:
`sink(MagicMock(), None)` and this direct foreign-database call never go through `Database.get_raw_connection -> notify_cursor(cursor, self)`, so a regression that rejects every non-None database, or drops the `self` forwarding in `models/core.py`, would leave all of these tests green while cancellation stops working for real chart queries. Could a test drive `notify_cursor(cursor, database)` with the task's own database and assert the handler registers once, and with another database and assert it doesn't?
##########
superset/common/query_context_processor.py:
##########
@@ -358,6 +362,39 @@ def get_df_payload_result(
# nonce reads the freshly-cached result instead of recomputing it.
self._mark_force_executed(query_obj, cache_key, cache.result_persisted)
+ # Annotation data is fetched per requesting user (and, for chart-backed
+ # layers, scoped by the referenced chart datasource's RLS), so it is
+ # resolved and cached under its own entry β independent of whether the
+ # (shareable) dataframe above was a hit or a miss β rather than forcing
+ # every viewer of the same chart onto their own full dataframe copy.
+ annotation_data: dict[str, Any] = {}
+ if (
+ query_obj
+ and query_obj.annotation_layers
+ and cache.status != QueryStatus.FAILED
+ ):
+ annotation_start_ns = time.perf_counter_ns()
+ try:
+ annotation_data = self._get_annotation_data_cached(
Review Comment:
With the dataframe key no longer scoped by viewer, a second viewer can join the first viewer's SHARED chart-data task, but that task still runs the annotation fetch under the submitter's access. If the submitter's annotation fetch raises `SemanticResultCompletenessError`, or fails validation inside a contribution totals prerequisite (`is_totals_prerequisite` re-raises a FAILED payload in `execute_chart_query`), the task fails for the joined viewer too. For example, viewer A lacks `can_read Annotation` while viewer B has it and a warm annotation entry; both force-refresh the same expired chart, B joins A's task, and B gets a task failure even though B's own execution would have succeeded. Should annotation failures be kept out of the dataframe-keyed task outcome, or the task identity stay split by annotation scope?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] fix(charts): decouple annotation cache from the shared dataframe cache [superset]
Posted by "sadpandajoe (via GitHub)" <gi...@apache.org>.
sadpandajoe commented on code in PR #44406:
URL: https://github.com/apache/superset/pull/44406#discussion_r4329621805
##########
tests/unit_tests/common/test_query_context_processor.py:
##########
@@ -3167,11 +3915,211 @@ def payload(context: QueryContext, **kwargs: Any) -> dict[str, Any]:
datasource="native-1", annotation_context=context
)
assert current_key != legacy_key
- assert context == {
- **legacy_context,
- "semantic_filter_protocol": "semantic-null-filters-v1",
- "source_versions": {"7": view.metadata_generation},
- }
+ assert context is not None
+ assert context["source_scope"][0]["semantic_filter_protocol"] == (
+ "semantic-null-filters-v1"
+ )
+ assert context["source_scope"][0]["metadata_version"] == view.metadata_generation
+
+
+def _annotated_payload_fixture(
+ *, status: str = "success"
+) -> tuple[QueryContextProcessor, Any, MagicMock]:
+ """A processor, annotated query object, and a dataframe cache hit."""
+ from superset.common.query_object import QueryObject
+
+ mock_query_context = MagicMock()
+ mock_query_context.force = False
+ mock_datasource = MagicMock()
+ mock_datasource.column_names = ["col1"]
+ processor = QueryContextProcessor(mock_query_context)
+ processor._qc_datasource = mock_datasource
+ query_obj = QueryObject(
+ datasource=mock_datasource,
+ columns=["col1"],
+ annotation_layers=[
+ {
+ "annotationType": "EVENT",
+ "sourceType": "NATIVE",
+ "name": "a",
+ "value": 1,
+ }
+ ],
+ )
+ cache = MagicMock()
+ cache.is_loaded = True
+ cache.applied_filter_columns = ["col1"]
+ cache.df = pd.DataFrame({"col1": [1, 2, 3]})
+ cache.query = ""
+ cache.status = status
+ cache.error_message = "dataframe failed" if status == QueryStatus.FAILED else None
+ cache.is_cached = True
+ cache.applied_template_filters = []
+ cache.rejected_filter_columns = []
+ cache.bq_memory_limited = False
+ cache.result_persisted = False
+ return processor, query_obj, cache
+
+
+def _run_annotated_payload(
+ processor: QueryContextProcessor,
+ query_obj: Any,
+ cache: MagicMock,
+ *,
+ force_cached: bool = False,
+) -> Any:
+ with (
+ patch(
+ "superset.common.query_context_processor.QueryCacheManager"
+ ) as mock_cache_manager,
+ patch.object(query_obj, "validate", return_value=None),
+ patch.object(processor, "query_cache_key", return_value="df-key"),
+ patch.object(processor, "annotation_cache_key", return_value="ann-key"),
+ patch.object(processor, "get_cache_timeout", return_value=3600),
+ ):
+ mock_cache_manager.get.return_value = cache
+ return processor.get_df_payload_result(query_obj, force_cached=force_cached)
+
+
+def test_get_df_payload_result_skips_annotations_when_dataframe_failed() -> None:
+ """A failed dataframe keeps its own error and never triggers the fetch."""
+ processor, query_obj, cache = _annotated_payload_fixture(status=QueryStatus.FAILED)
+ with patch.object(processor, "_get_annotation_data_cached") as mock_annotation:
+ result = _run_annotated_payload(processor, query_obj, cache)
+ mock_annotation.assert_not_called()
+ assert result.payload["annotation_data"] == {}
+ assert result.payload["error"] == "dataframe failed"
+
+
+def test_get_df_payload_result_force_cached_misses_without_annotation_entry() -> None:
+ """Through the real helper, ``force_cached`` with no annotation entry
+ raises ``CacheLoadError`` instead of running the annotation SQL."""
+ processor, query_obj, cache = _annotated_payload_fixture()
+ with (
+ patch("superset.common.query_context_processor.cache_manager") as cm,
+ patch.object(processor, "get_annotation_data") as mock_live,
+ ):
+ cm.data_cache.get.return_value = None
+ with pytest.raises(CacheLoadError):
+ _run_annotated_payload(processor, query_obj, cache, force_cached=True)
+ mock_live.assert_not_called()
+
+
+def test_get_df_payload_result_annotation_failure_marks_payload_failed() -> None:
+ """An annotation error on a good dataframe fails the payload."""
+ processor, query_obj, cache = _annotated_payload_fixture()
+ with (
+ patch("superset.common.query_context_processor.cache_manager") as cm,
+ patch.object(
+ processor,
+ "get_annotation_data",
+ side_effect=QueryObjectValidationError("annotation boom"),
+ ),
+ ):
+ cm.data_cache.get.return_value = None
+ result = _run_annotated_payload(processor, query_obj, cache)
+ assert cache.status == QueryStatus.FAILED
+ assert result.payload["error"] == "annotation boom"
+
+
+def test_get_df_payload_result_completeness_error_escapes_annotation_fetch() -> None:
+ """A semantic annotation's incomplete-result signal must propagate (as the
+ dataframe path does) so the async task is not reported successful."""
+ from superset.exceptions import SemanticResultCompletenessError
+
+ processor, query_obj, cache = _annotated_payload_fixture()
+ with (
+ patch("superset.common.query_context_processor.cache_manager") as cm,
+ patch.object(
+ processor,
+ "get_annotation_data",
+ side_effect=SemanticResultCompletenessError("unverified"),
+ ),
+ ):
+ cm.data_cache.get.return_value = None
+ with pytest.raises(SemanticResultCompletenessError):
+ _run_annotated_payload(processor, query_obj, cache)
+
+
+def test_get_df_payload_result_times_annotation_fetch_as_data_acquisition() -> None:
+ """Annotation work lands in ``data_acquisition_ns`` even on a dataframe
+ cache hit (which otherwise reports no acquisition time)."""
+ processor, query_obj, cache = _annotated_payload_fixture()
+
+ def slow_annotations(*_: Any, **__: Any) -> dict[str, Any]:
+ time.sleep(0.01)
+ return {}
+
+ with patch.object(
+ processor, "_get_annotation_data_cached", side_effect=slow_annotations
+ ):
+ result = _run_annotated_payload(processor, query_obj, cache)
+ assert result.timing.data_acquisition_ns is not None
+ assert result.timing.data_acquisition_ns >= 10_000_000
Review Comment:
The new timing test only runs with a dataframe cache hit, so the summing path is never exercised: if the annotation duration replaced the dataframe time instead of being added to it, a cold annotated chart would report only the annotation time as `data_acquisition_ms` and every assertion here would still pass. Could you add a case where both the dataframe fetch and the annotation fetch run with a deterministic clock (say 30 ns and 20 ns) and assert `data_acquisition_ns == 50`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] fix(charts): decouple annotation cache from the shared dataframe cache [superset]
Posted by "rusackas (via GitHub)" <gi...@apache.org>.
rusackas commented on code in PR #44406:
URL: https://github.com/apache/superset/pull/44406#discussion_r4338870176
##########
superset/common/query_context_processor.py:
##########
@@ -478,73 +518,256 @@ def query_cache_key(self, query_obj: QueryObject, **kwargs: Any) -> str | None:
)
return cache_key
- def _annotation_cache_context(self, query_obj: QueryObject) -> dict[str, Any]:
+ def annotation_cache_key(self, query_obj: QueryObject) -> str | None:
"""
- Cache-key material binding cached annotation data to its security
- context.
-
- Annotation payloads are fetched per requesting user and stored on the
- same cache entry as the dataframe, so the key also binds the requesting
- user and, for chart-backed layers, the RLS clauses of the referenced
- chart's datasource.
+ Cache key for this query's annotation-layer payload, or ``None`` when
+ the query has no annotation layers or when some layer's access scope
+ could not be resolved (the payload is then never cached, see
+ :meth:`_annotation_source_scope`).
+
+ Annotation payloads are fetched under the requesting user's access
+ scope, which is a stricter security requirement than the dataframe
+ itself has. Keying them separately from :meth:`query_cache_key` keeps
+ that scoping from forcing every distinct viewer of an annotated chart
+ onto their own full copy of the (potentially much larger) shared
+ dataframe: users with the same access scope share this key too.
+ """
+ if not query_obj or not query_obj.annotation_layers:
+ return None
+ if (annotation_context := self._annotation_cache_context(query_obj)) is None:
+ return None
+ return self.query_cache_key(query_obj, annotation_context=annotation_context)
+
+ def _annotation_cache_context(
+ self, query_obj: QueryObject
+ ) -> dict[str, Any] | None:
+ """
+ Cache-key material binding annotation data to its security *scope* so
+ users with the same access share a cache entry and users with a
+ different scope β or no access β never read each other's data.
+
+ * NATIVE layers: the ``can_read`` permission on ``Annotation``, the
+ only user-dependent dimension of these global records.
+ * Chart-backed (``line``/``table``) layers: see
+ :meth:`_annotation_source_scope`, recorded per *layer* (not per
+ referenced chart) since two layers can reference the same chart
+ with different overrides, and so different scopes.
+
+ Returns ``None`` when any layer's scope cannot be resolved, meaning
+ the payload must not be cached at all.
"""
- source_rls: dict[str, list[str] | None] = {}
- source_versions: dict[str, tuple[str, str]] = {}
- metadata_versions: dict[str, str] = {}
+ context: dict[str, Any] = {}
+
+ if any(
+ layer.get("sourceType") == "NATIVE" for layer in query_obj.annotation_layers
+ ):
+ context["annotation_read"] = security_manager.can_access(
+ "can_read", "Annotation"
+ )
+
+ source_scope: list[dict[str, Any]] = []
for layer in query_obj.annotation_layers:
if (
layer.get("sourceType")
not in ANNOTATION_SOURCE_TYPES_WITH_CHART_REFERENCE
):
continue
- layer_value = layer.get("value")
+ if (scope := self._annotation_source_scope(layer)) is None:
+ return None
+ source_scope.append({"value": str(layer.get("value")), **scope})
+ if source_scope:
+ context["source_scope"] = source_scope
+
+ return context
+
+ def _annotation_source_scope(self, layer: dict[str, Any]) -> dict[str, Any] | None:
+ """
+ Access and data-identity cache-key material for one chart-backed
+ annotation layer, or ``None`` when it cannot be resolved.
+
+ ``None`` means "do not cache": a missing chart, a chart with neither
+ a saved query context nor a resolvable datasource, or any error while
+ deriving the scope would otherwise collapse an authorized and a
+ denied requester onto one shared identity, letting the latter read
+ the former's warmed payload. The caller fetches live instead. A
+ referenced chart whose own caching is disabled
+ (``CACHE_DISABLED_TIMEOUT``) is ``None`` too: its rows are never
+ shared through its own cache, so they must not be shared through the
+ host chart's annotation entry either.
+
+ ``access`` keeps a user denied the referenced chart's datasource from
+ reading an authorized user's cached payload. When the chart has a
+ saved query context, this runs the *same* authorization path
+ :meth:`get_viz_annotation_data` executes
+ (``QueryContext.raise_for_access``) rather than the coarser
+ :meth:`SecurityManager.can_access_datasource` β a requester whose
+ access comes from a dashboard/viewer-promiscuous-mode bypass (which
+ depends on the chart's own saved ``form_data``, e.g. its
+ ``slice_id``/``dashboardId``) would otherwise still fail that coarser,
+ context-free check and collapse onto the same denied scope as a
+ genuinely unauthorized requester, letting the latter read the
+ former's cached payload. ``data_key`` is the annotation chart's own
+ query cache key(s) β derived from the same, override-applied and
+ result-type-prepared query objects actually executed (see
+ :meth:`_apply_annotation_overrides` and
+ :func:`~superset.common.query_actions.prepare_query_object`), so it
+ captures the datasource version, RLS clauses, and any per-user
+ Jinja/virtual-dataset RLS material exactly as the live fetch would,
+ including material an override only introduces at a finer grain or
+ that a samples query's cleared metrics only expose at execution.
+ Reusing this logic avoids re-deriving it and automatically inherits
+ any future correctness fixes made there.
+ """
+ layer_value = layer.get("value")
+ try:
chart = (
ChartDAO.find_by_id(layer_value) if layer_value is not None else None
)
- annotation_datasource: Datasource | None = (
- chart.resolved_datasource if chart else None
- )
- if chart and isinstance(chart.query_context, str):
+ if chart is None:
+ return None
+
+ # The saved query context is what the live fetch executes and
+ # authorizes against, so it is consulted first, independent of
+ # the chart's own datasource pointer: a chart can have its
+ # ``datasource_id`` cleared while its saved query context still
+ # targets a valid dataset, and the live fetch would still work.
+ annotation_query_context = chart.get_query_context()
+ if annotation_query_context is not None:
+ if (
+ annotation_query_context.get_cache_timeout()
+ == CACHE_DISABLED_TIMEOUT
+ ):
+ return None
+ self._apply_annotation_overrides(annotation_query_context, layer)
try:
- source: dict[str, Any] = json.loads(chart.query_context)[
- "datasource"
+ annotation_query_context.raise_for_access()
+ access: Any = True
+ except SupersetSecurityException:
+ access = False
+ # A denied requester never reads or writes a cached payload,
+ # so skip deriving a data identity (which can render the
+ # dataset's Jinja) for a resource they cannot access.
+ # Keyed on the query object the fetch will execute: a samples
+ # or drill-detail result type rewrites the saved query (e.g.
+ # clears its metrics) before running it, and that can change
+ # the per-user Jinja material the key has to carry.
+ data_key: Any = (
+ [
+ annotation_query_context.query_cache_key(
+ prepare_query_object(annotation_query_context, query_object)
Review Comment:
Good catch. The key now passes `query_object.result_type or annotation_query_context.result_type` to `prepare_query_object`, matching what execution uses. Pushed in 79d98f4.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] fix(charts): decouple annotation cache from the shared dataframe cache [superset]
Posted by "sadpandajoe (via GitHub)" <gi...@apache.org>.
sadpandajoe commented on code in PR #44406:
URL: https://github.com/apache/superset/pull/44406#discussion_r4075824619
##########
superset/common/query_context_processor.py:
##########
@@ -445,31 +464,150 @@ def query_cache_key(self, query_obj: QueryObject, **kwargs: Any) -> str | None:
)
return cache_key
- def _annotation_cache_context(self, query_obj: QueryObject) -> dict[str, Any]:
+ def annotation_cache_key(self, query_obj: QueryObject) -> str | None:
+ """
+ Cache key for this query's annotation-layer payload, or ``None`` when
+ the query has no annotation layers.
+
+ Annotation payloads are fetched under the requesting user's access
+ scope, which is a stricter security requirement than the dataframe
+ itself has. Keying them separately from :meth:`query_cache_key` keeps
+ that scoping from forcing every distinct viewer of an annotated chart
+ onto their own full copy of the (potentially much larger) shared
+ dataframe: users with the same access scope share this key too.
"""
- Cache-key material binding cached annotation data to its security
- context.
+ if not query_obj or not query_obj.annotation_layers:
+ return None
+ return self.query_cache_key(
+ query_obj, annotation_context=self._annotation_cache_context(query_obj)
+ )
- Annotation payloads are fetched per requesting user and stored on the
- same cache entry as the dataframe, so the key also binds the requesting
- user and, for chart-backed layers, the RLS clauses of the referenced
- chart's datasource.
+ def _annotation_cache_context(self, query_obj: QueryObject) -> dict[str, Any]:
+ """
+ Cache-key material binding annotation data to its security *scope* so
+ users with the same access share a cache entry and users with a
+ different scope β or no access β never read each other's data.
+
+ * NATIVE layers: the ``can_read`` permission on ``Annotation``, the
+ only user-dependent dimension of these global records.
+ * Chart-backed (``line``/``table``) layers: see
+ :meth:`_annotation_source_scope`.
"""
- source_rls: dict[str, list[str] | None] = {}
+ context: dict[str, Any] = {}
+
+ if any(
+ layer.get("sourceType") == "NATIVE" for layer in query_obj.annotation_layers
+ ):
+ context["annotation_read"] = security_manager.can_access(
+ "can_read", "Annotation"
+ )
+
+ source_scope: dict[str, Any] = {}
for layer in query_obj.annotation_layers:
if layer.get("sourceType") not in ("line", "table"):
continue
layer_value = layer.get("value")
- chart = (
- ChartDAO.find_by_id(layer_value) if layer_value is not None else None
+ source_scope[str(layer_value)] = self._annotation_source_scope(layer_value)
+ if source_scope:
+ context["source_scope"] = source_scope
+
+ return context
+
+ def _annotation_source_scope(self, layer_value: Any) -> dict[str, Any]:
+ """
+ Access and data-identity cache-key material for one chart-backed
+ annotation layer.
+
+ ``access`` keeps a user denied the referenced chart's datasource from
+ reading an authorized user's cached payload. ``data_key`` is the
+ annotation chart's own query cache key(s), which already capture the
+ datasource version, RLS clauses, and any per-user Jinja/virtual-dataset
+ RLS material β reusing it here avoids re-deriving that logic and
+ automatically inherits any future correctness fixes made there.
+ """
+ chart = ChartDAO.find_by_id(layer_value) if layer_value is not None else None
+ datasource = chart.datasource if chart else None
Review Comment:
Semantic-view charts have `chart.datasource == None`, so this gives every requester the same `{access: None, data_key: None}` annotation key. An authorized user can warm that entry and a user without the semantic-view grant then receives its cached payload before validation runs. Should this use `chart.resolved_datasource` and treat an unresolved source as inaccessible?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] fix(charts): decouple annotation cache from the shared dataframe cache [superset]
Posted by "codecov[bot] (via GitHub)" <gi...@apache.org>.
codecov[bot] commented on PR #44406:
URL: https://github.com/apache/superset/pull/44406#issuecomment-5722407865
## [Codecov](https://app.codecov.io/gh/apache/superset/pull/44406?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:x: Patch coverage is `78.57143%` with `6 lines` in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 76.62%. Comparing base ([`22ae440`](https://app.codecov.io/gh/apache/superset/commit/22ae4405d4eb033a3c477e898c6431cb6b488dca?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`19954fd`](https://app.codecov.io/gh/apache/superset/commit/19954fdb004845b9cf25f7ffe0ccb97b8f7211df?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
| [Files with missing lines](https://app.codecov.io/gh/apache/superset/pull/44406?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [superset/common/query\_context\_processor.py](https://app.codecov.io/gh/apache/superset/pull/44406?src=pr&el=tree&filepath=superset%2Fcommon%2Fquery_context_processor.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-c3VwZXJzZXQvY29tbW9uL3F1ZXJ5X2NvbnRleHRfcHJvY2Vzc29yLnB5) | 78.57% | [6 Missing :warning: ](https://app.codecov.io/gh/apache/superset/pull/44406?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #44406 +/- ##
===========================================
+ Coverage 57.37% 76.62% +19.24%
===========================================
Files 2932 2932
Lines 174781 174806 +25
Branches 40594 40598 +4
===========================================
+ Hits 100285 133949 +33664
+ Misses 73539 38092 -35447
- Partials 957 2765 +1808
```
| [Flag](https://app.codecov.io/gh/apache/superset/pull/44406/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Ξ | |
|---|---|---|
| [hive](https://app.codecov.io/gh/apache/superset/pull/44406/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `37.29% <7.14%> (-0.01%)` | :arrow_down: |
| [presto](https://app.codecov.io/gh/apache/superset/pull/44406/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `39.17% <17.85%> (-0.02%)` | :arrow_down: |
| [python](https://app.codecov.io/gh/apache/superset/pull/44406/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `77.29% <78.57%> (+38.05%)` | :arrow_up: |
| [unit](https://app.codecov.io/gh/apache/superset/pull/44406/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `76.42% <78.57%> (?)` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
</details>
[:umbrella: View full report in Codecov by Harness](https://app.codecov.io/gh/apache/superset/pull/44406?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
- :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] fix(charts): decouple annotation cache from the shared dataframe cache [superset]
Posted by "rusackas (via GitHub)" <gi...@apache.org>.
rusackas commented on code in PR #44406:
URL: https://github.com/apache/superset/pull/44406#discussion_r4151621247
##########
superset/common/query_context_processor.py:
##########
@@ -448,31 +479,167 @@ def query_cache_key(self, query_obj: QueryObject, **kwargs: Any) -> str | None:
)
return cache_key
- def _annotation_cache_context(self, query_obj: QueryObject) -> dict[str, Any]:
+ def annotation_cache_key(self, query_obj: QueryObject) -> str | None:
+ """
+ Cache key for this query's annotation-layer payload, or ``None`` when
+ the query has no annotation layers.
+
+ Annotation payloads are fetched under the requesting user's access
+ scope, which is a stricter security requirement than the dataframe
+ itself has. Keying them separately from :meth:`query_cache_key` keeps
+ that scoping from forcing every distinct viewer of an annotated chart
+ onto their own full copy of the (potentially much larger) shared
+ dataframe: users with the same access scope share this key too.
"""
- Cache-key material binding cached annotation data to its security
- context.
+ if not query_obj or not query_obj.annotation_layers:
+ return None
+ return self.query_cache_key(
+ query_obj, annotation_context=self._annotation_cache_context(query_obj)
+ )
- Annotation payloads are fetched per requesting user and stored on the
- same cache entry as the dataframe, so the key also binds the requesting
- user and, for chart-backed layers, the RLS clauses of the referenced
- chart's datasource.
+ def _annotation_cache_context(self, query_obj: QueryObject) -> dict[str, Any]:
+ """
+ Cache-key material binding annotation data to its security *scope* so
+ users with the same access share a cache entry and users with a
+ different scope β or no access β never read each other's data.
+
+ * NATIVE layers: the ``can_read`` permission on ``Annotation``, the
+ only user-dependent dimension of these global records.
+ * Chart-backed (``line``/``table``) layers: see
+ :meth:`_annotation_source_scope`.
"""
- source_rls: dict[str, list[str] | None] = {}
+ context: dict[str, Any] = {}
+
+ if any(
+ layer.get("sourceType") == "NATIVE" for layer in query_obj.annotation_layers
+ ):
+ context["annotation_read"] = security_manager.can_access(
+ "can_read", "Annotation"
+ )
+
+ source_scope: dict[str, Any] = {}
for layer in query_obj.annotation_layers:
if layer.get("sourceType") not in ("line", "table"):
continue
layer_value = layer.get("value")
+ source_scope[str(layer_value)] = self._annotation_source_scope(layer_value)
+ if source_scope:
+ context["source_scope"] = source_scope
+
+ return context
+
+ def _annotation_source_scope(self, layer_value: Any) -> dict[str, Any]:
+ """
+ Access and data-identity cache-key material for one chart-backed
+ annotation layer.
+
+ ``access`` keeps a user denied the referenced chart's datasource from
+ reading an authorized user's cached payload. ``data_key`` is the
+ annotation chart's own query cache key(s), which already capture the
+ datasource version, RLS clauses, and any per-user Jinja/virtual-dataset
+ RLS material β reusing it here avoids re-deriving that logic and
+ automatically inherits any future correctness fixes made there.
+ """
+ datasource = None
+ try:
chart = (
ChartDAO.find_by_id(layer_value) if layer_value is not None else None
)
- annotation_datasource = chart.datasource if chart else None
- source_rls[str(layer.get("value"))] = (
- security_manager.get_rls_cache_key(annotation_datasource)
- if annotation_datasource
- else None
+ # resolved_datasource, not datasource: the latter is pinned to
+ # table-backed datasources and resolves to None for a
+ # semantic-view-backed chart, which would otherwise collapse
+ # every requester onto the same {access: None, data_key: None}
+ # scope below regardless of their actual access.
+ datasource = chart.resolved_datasource if chart else None
+ if chart is None or datasource is None:
+ return {"access": None, "data_key": None}
+
+ access = security_manager.can_access_datasource(datasource)
Review Comment:
Good catch. Access now runs through the referenced chart's own `raise_for_access()` instead of `can_access_datasource`, so a promiscuous-mode viewer bypass gets its own scope instead of collapsing onto the denied one. Pushed in d5fe17a4c28f.
##########
superset/common/query_context_processor.py:
##########
@@ -448,31 +479,167 @@ def query_cache_key(self, query_obj: QueryObject, **kwargs: Any) -> str | None:
)
return cache_key
- def _annotation_cache_context(self, query_obj: QueryObject) -> dict[str, Any]:
+ def annotation_cache_key(self, query_obj: QueryObject) -> str | None:
+ """
+ Cache key for this query's annotation-layer payload, or ``None`` when
+ the query has no annotation layers.
+
+ Annotation payloads are fetched under the requesting user's access
+ scope, which is a stricter security requirement than the dataframe
+ itself has. Keying them separately from :meth:`query_cache_key` keeps
+ that scoping from forcing every distinct viewer of an annotated chart
+ onto their own full copy of the (potentially much larger) shared
+ dataframe: users with the same access scope share this key too.
"""
- Cache-key material binding cached annotation data to its security
- context.
+ if not query_obj or not query_obj.annotation_layers:
+ return None
+ return self.query_cache_key(
+ query_obj, annotation_context=self._annotation_cache_context(query_obj)
+ )
- Annotation payloads are fetched per requesting user and stored on the
- same cache entry as the dataframe, so the key also binds the requesting
- user and, for chart-backed layers, the RLS clauses of the referenced
- chart's datasource.
+ def _annotation_cache_context(self, query_obj: QueryObject) -> dict[str, Any]:
+ """
+ Cache-key material binding annotation data to its security *scope* so
+ users with the same access share a cache entry and users with a
+ different scope β or no access β never read each other's data.
+
+ * NATIVE layers: the ``can_read`` permission on ``Annotation``, the
+ only user-dependent dimension of these global records.
+ * Chart-backed (``line``/``table``) layers: see
+ :meth:`_annotation_source_scope`.
"""
- source_rls: dict[str, list[str] | None] = {}
+ context: dict[str, Any] = {}
+
+ if any(
+ layer.get("sourceType") == "NATIVE" for layer in query_obj.annotation_layers
+ ):
+ context["annotation_read"] = security_manager.can_access(
+ "can_read", "Annotation"
+ )
+
+ source_scope: dict[str, Any] = {}
for layer in query_obj.annotation_layers:
if layer.get("sourceType") not in ("line", "table"):
continue
layer_value = layer.get("value")
+ source_scope[str(layer_value)] = self._annotation_source_scope(layer_value)
+ if source_scope:
+ context["source_scope"] = source_scope
+
+ return context
+
+ def _annotation_source_scope(self, layer_value: Any) -> dict[str, Any]:
+ """
+ Access and data-identity cache-key material for one chart-backed
+ annotation layer.
+
+ ``access`` keeps a user denied the referenced chart's datasource from
+ reading an authorized user's cached payload. ``data_key`` is the
+ annotation chart's own query cache key(s), which already capture the
+ datasource version, RLS clauses, and any per-user Jinja/virtual-dataset
+ RLS material β reusing it here avoids re-deriving that logic and
+ automatically inherits any future correctness fixes made there.
+ """
+ datasource = None
+ try:
chart = (
ChartDAO.find_by_id(layer_value) if layer_value is not None else None
)
- annotation_datasource = chart.datasource if chart else None
- source_rls[str(layer.get("value"))] = (
- security_manager.get_rls_cache_key(annotation_datasource)
- if annotation_datasource
- else None
+ # resolved_datasource, not datasource: the latter is pinned to
+ # table-backed datasources and resolves to None for a
+ # semantic-view-backed chart, which would otherwise collapse
+ # every requester onto the same {access: None, data_key: None}
+ # scope below regardless of their actual access.
+ datasource = chart.resolved_datasource if chart else None
+ if chart is None or datasource is None:
+ return {"access": None, "data_key": None}
+
+ access = security_manager.can_access_datasource(datasource)
+ # Fall back to the RLS-clause identity when the chart has no saved
+ # query context to key on.
+ annotation_query_context = chart.get_query_context()
+ data_key: Any = (
+ [
+ annotation_query_context.query_cache_key(query_object)
Review Comment:
Good catch. The override now gets applied to the referenced query context before the cache key is derived too, not just before it runs, so a grain-dependent RLS clause gets captured. Pushed in d5fe17a4c28f.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] fix(charts): decouple annotation cache from the shared dataframe cache [superset]
Posted by "sadpandajoe (via GitHub)" <gi...@apache.org>.
sadpandajoe commented on code in PR #44406:
URL: https://github.com/apache/superset/pull/44406#discussion_r4408362968
##########
superset/common/query_context_processor.py:
##########
@@ -480,96 +521,282 @@ def query_cache_key(self, query_obj: QueryObject, **kwargs: Any) -> str | None:
)
return cache_key
- def _annotation_cache_context(self, query_obj: QueryObject) -> dict[str, Any]:
+ def annotation_cache_key(self, query_obj: QueryObject) -> str | None:
"""
- Cache-key material binding cached annotation data to its security
- context.
-
- Annotation payloads are fetched per requesting user and stored on the
- same cache entry as the dataframe, so the key also binds the requesting
- user and, for chart-backed layers, the RLS clauses of the referenced
- chart's datasource, including those injected into the tables its SQL
- reads. The user id cannot tell embedded guests apart, so for them the
- RLS is the only thing that does.
+ Cache key for this query's annotation-layer payload, or ``None`` when
+ the query has no annotation layers or when some layer's access scope
+ could not be resolved (the payload is then never cached, see
+ :meth:`_annotation_source_scope`).
+
+ Annotation payloads are fetched under the requesting user's access
+ scope, which is a stricter security requirement than the dataframe
+ itself has. Keying them separately from :meth:`query_cache_key` keeps
+ that scoping from forcing every distinct viewer of an annotated chart
+ onto their own full copy of the (potentially much larger) shared
+ dataframe: users with the same access scope share this key too.
"""
- source_rls: dict[str, list[str] | None] = {}
- source_inner_rls: dict[str, list[str]] = {}
- source_rendered_rls: dict[str, list[str]] = {}
- source_versions: dict[str, tuple[str, str]] = {}
- metadata_versions: dict[str, str] = {}
+ if not query_obj or not query_obj.annotation_layers:
+ return None
+ if (annotation_context := self._annotation_cache_context(query_obj)) is None:
+ return None
+ return self.query_cache_key(query_obj, annotation_context=annotation_context)
+
+ def _annotation_cache_context(
+ self, query_obj: QueryObject
+ ) -> dict[str, Any] | None:
+ """
+ Cache-key material binding annotation data to its security *scope* so
+ users with the same access share a cache entry and users with a
+ different scope β or no access β never read each other's data.
+
+ * NATIVE layers: the ``can_read`` permission on ``Annotation``, the
+ only user-dependent dimension of these global records.
+ * Chart-backed (``line``/``table``) layers: see
+ :meth:`_annotation_source_scope`, recorded per *layer* (not per
+ referenced chart) since two layers can reference the same chart
+ with different overrides, and so different scopes.
+
+ Returns ``None`` when any layer's scope cannot be resolved, meaning
+ the payload must not be cached at all.
+ """
+ context: dict[str, Any] = {}
+
+ if any(
+ layer.get("sourceType") == "NATIVE" for layer in query_obj.annotation_layers
+ ):
+ context["annotation_read"] = security_manager.can_access(
+ "can_read", "Annotation"
+ )
+
+ source_scope: list[dict[str, Any]] = []
for layer in query_obj.annotation_layers:
if (
layer.get("sourceType")
not in ANNOTATION_SOURCE_TYPES_WITH_CHART_REFERENCE
):
continue
- layer_value = layer.get("value")
- annotation_datasource: Datasource | None = (
- self._annotation_source_datasource(layer_value)
+ if (scope := self._annotation_source_scope(layer)) is None:
+ return None
+ source_scope.append({"value": str(layer.get("value")), **scope})
+ if source_scope:
+ context["source_scope"] = source_scope
+
+ return context
+
+ def _annotation_source_scope(self, layer: dict[str, Any]) -> dict[str, Any] | None:
+ """
+ Access and data-identity cache-key material for one chart-backed
+ annotation layer, or ``None`` when it cannot be resolved.
+
+ ``None`` means "do not cache": a missing chart, a chart with neither
+ a saved query context nor a resolvable datasource, or any error while
+ deriving the scope would otherwise collapse an authorized and a
+ denied requester onto one shared identity, letting the latter read
+ the former's warmed payload. The caller fetches live instead. A
+ referenced chart whose own caching is disabled
+ (``CACHE_DISABLED_TIMEOUT``) is ``None`` too: its rows are never
+ shared through its own cache, so they must not be shared through the
+ host chart's annotation entry either.
+
+ ``access`` keeps a user denied the referenced chart's datasource from
+ reading an authorized user's cached payload. When the chart has a
+ saved query context, this runs the *same* authorization path
+ :meth:`get_viz_annotation_data` executes
+ (``QueryContext.raise_for_access``) rather than the coarser
+ :meth:`SecurityManager.can_access_datasource` β a requester whose
+ access comes from a dashboard/viewer-promiscuous-mode bypass (which
+ depends on the chart's own saved ``form_data``, e.g. its
+ ``slice_id``/``dashboardId``) would otherwise still fail that coarser,
+ context-free check and collapse onto the same denied scope as a
+ genuinely unauthorized requester, letting the latter read the
+ former's cached payload. ``data_key`` is the annotation chart's own
+ query cache key(s) β derived from the same, override-applied and
+ result-type-prepared query objects actually executed (see
+ :meth:`_apply_annotation_overrides` and
+ :func:`~superset.common.query_actions.prepare_query_object`), so it
+ captures the datasource version, RLS clauses, and any per-user
+ Jinja/virtual-dataset RLS material exactly as the live fetch would,
+ including material an override only introduces at a finer grain or
+ that a samples query's cleared metrics only expose at execution.
+ Reusing this logic avoids re-deriving it and automatically inherits
+ any future correctness fixes made there.
+ """
+ layer_value = layer.get("value")
+ try:
+ chart = (
+ ChartDAO.find_by_id(layer_value) if layer_value is not None else None
)
- if isinstance(annotation_datasource, SemanticView):
- metadata_versions[str(layer_value)] = (
- annotation_datasource.metadata_generation
+ if chart is None:
+ return None
+
+ # The saved query context is what the live fetch executes and
+ # authorizes against, so it is consulted first, independent of
+ # the chart's own datasource pointer: a chart can have its
+ # ``datasource_id`` cleared while its saved query context still
+ # targets a valid dataset, and the live fetch would still work.
+ annotation_query_context = chart.get_query_context()
+ if annotation_query_context is not None:
+ if (
+ annotation_query_context.get_cache_timeout()
+ == CACHE_DISABLED_TIMEOUT
+ ):
+ return None
+ self._apply_annotation_overrides(annotation_query_context, layer)
+ try:
+ annotation_query_context.raise_for_access()
+ access: Any = True
+ except SupersetSecurityException:
+ access = False
+ # A denied requester never reads or writes a cached payload,
+ # so skip deriving a data identity (which can render the
+ # dataset's Jinja) for a resource they cannot access.
+ # Keyed on the query object the fetch will execute: a samples
+ # or drill-detail result type rewrites the saved query (e.g.
+ # clears its metrics) before running it, and that can change
+ # the per-user Jinja material the key has to carry.
+ data_key: Any = (
+ [
+ annotation_query_context.query_cache_key(
Review Comment:
A saved-context annotation source keys only on the source chart's query cache key, which drops the rendered RLS fingerprint the old per-layer key carried. The extra-cache-key material is deduplicated and sorted, so with source-dataset RLS like `owner = '{{ current_username() }}' AND region = '{{ url_param("region") }}'`, user A with `?region=us` and user B with `?region=eu` (or any permutation of the same values) derive the same `data_key`, `access`, and annotation key. Once A warms the host's annotation entry and the shorter-lived source cache expires, B is served A's rows. Should the saved-context branch also fold in `_datasource_rls_data_key(annotation_query_context.datasource)` so the rendered and inner RLS are part of the scope here too?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] fix(charts): decouple annotation cache from the shared dataframe cache [superset]
Posted by "rusackas (via GitHub)" <gi...@apache.org>.
rusackas commented on code in PR #44406:
URL: https://github.com/apache/superset/pull/44406#discussion_r4118924906
##########
superset/common/query_context_processor.py:
##########
@@ -328,6 +327,26 @@ def get_df_payload_result(
# nonce reads the freshly-cached result instead of recomputing it.
self._mark_force_executed(query_obj, cache_key, cache.result_persisted)
+ # Annotation data is fetched per requesting user (and, for chart-backed
+ # layers, scoped by the referenced chart datasource's RLS), so it is
+ # resolved and cached under its own entry β independent of whether the
+ # (shareable) dataframe above was a hit or a miss β rather than forcing
+ # every viewer of the same chart onto their own full dataframe copy.
+ annotation_data: dict[str, Any] = {}
+ if query_obj and annotation_key and cache.status != QueryStatus.FAILED:
+ try:
+ annotation_data = self._get_annotation_data_cached(
+ query_obj=query_obj,
+ cache_key=annotation_key,
+ force_query=force_query,
Review Comment:
Confirmed and actually fixed this time (traced it: my earlier reply here claimed a fix that never landed in the code β force_query was still the shared dataframe value at the annotation call site). Pushed 604acb6895: the annotation cache now resolves and records its own force-nonce marker against annotation_key, independent of the dataframe's. Added a regression test that sets the dataframe's marker but not the annotation's and asserts the annotation path still forces.
##########
superset/common/query_context_processor.py:
##########
@@ -445,31 +464,150 @@ def query_cache_key(self, query_obj: QueryObject, **kwargs: Any) -> str | None:
)
return cache_key
- def _annotation_cache_context(self, query_obj: QueryObject) -> dict[str, Any]:
+ def annotation_cache_key(self, query_obj: QueryObject) -> str | None:
+ """
+ Cache key for this query's annotation-layer payload, or ``None`` when
+ the query has no annotation layers.
+
+ Annotation payloads are fetched under the requesting user's access
+ scope, which is a stricter security requirement than the dataframe
+ itself has. Keying them separately from :meth:`query_cache_key` keeps
+ that scoping from forcing every distinct viewer of an annotated chart
+ onto their own full copy of the (potentially much larger) shared
+ dataframe: users with the same access scope share this key too.
"""
- Cache-key material binding cached annotation data to its security
- context.
+ if not query_obj or not query_obj.annotation_layers:
+ return None
+ return self.query_cache_key(
+ query_obj, annotation_context=self._annotation_cache_context(query_obj)
+ )
- Annotation payloads are fetched per requesting user and stored on the
- same cache entry as the dataframe, so the key also binds the requesting
- user and, for chart-backed layers, the RLS clauses of the referenced
- chart's datasource.
+ def _annotation_cache_context(self, query_obj: QueryObject) -> dict[str, Any]:
+ """
+ Cache-key material binding annotation data to its security *scope* so
+ users with the same access share a cache entry and users with a
+ different scope β or no access β never read each other's data.
+
+ * NATIVE layers: the ``can_read`` permission on ``Annotation``, the
+ only user-dependent dimension of these global records.
+ * Chart-backed (``line``/``table``) layers: see
+ :meth:`_annotation_source_scope`.
"""
- source_rls: dict[str, list[str] | None] = {}
+ context: dict[str, Any] = {}
+
+ if any(
+ layer.get("sourceType") == "NATIVE" for layer in query_obj.annotation_layers
+ ):
+ context["annotation_read"] = security_manager.can_access(
+ "can_read", "Annotation"
+ )
+
+ source_scope: dict[str, Any] = {}
for layer in query_obj.annotation_layers:
if layer.get("sourceType") not in ("line", "table"):
continue
layer_value = layer.get("value")
- chart = (
- ChartDAO.find_by_id(layer_value) if layer_value is not None else None
+ source_scope[str(layer_value)] = self._annotation_source_scope(layer_value)
+ if source_scope:
+ context["source_scope"] = source_scope
+
+ return context
+
+ def _annotation_source_scope(self, layer_value: Any) -> dict[str, Any]:
+ """
+ Access and data-identity cache-key material for one chart-backed
+ annotation layer.
+
+ ``access`` keeps a user denied the referenced chart's datasource from
+ reading an authorized user's cached payload. ``data_key`` is the
+ annotation chart's own query cache key(s), which already capture the
+ datasource version, RLS clauses, and any per-user Jinja/virtual-dataset
+ RLS material β reusing it here avoids re-deriving that logic and
+ automatically inherits any future correctness fixes made there.
+ """
+ chart = ChartDAO.find_by_id(layer_value) if layer_value is not None else None
+ datasource = chart.datasource if chart else None
Review Comment:
Same story as the other thread β the earlier reply claiming this was fixed wasn't accurate, `chart.datasource` was still there. Actually swapped it to `resolved_datasource` now (604acb6895), and added a test with `datasource = None` on the mock chart to make sure this doesn't regress silently again β before this fix it would've passed anyway since MagicMock's default `.datasource` is truthy.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] fix(charts): decouple annotation cache from the shared dataframe cache [superset]
Posted by "bito-code-review[bot] (via GitHub)" <gi...@apache.org>.
bito-code-review[bot] commented on code in PR #44406:
URL: https://github.com/apache/superset/pull/44406#discussion_r4043396474
##########
superset/common/query_context_processor.py:
##########
@@ -445,31 +464,144 @@ def query_cache_key(self, query_obj: QueryObject, **kwargs: Any) -> str | None:
)
return cache_key
- def _annotation_cache_context(self, query_obj: QueryObject) -> dict[str, Any]:
+ def annotation_cache_key(self, query_obj: QueryObject) -> str | None:
+ """
+ Cache key for this query's annotation-layer payload, or ``None`` when
+ the query has no annotation layers.
+
+ Annotation payloads are fetched under the requesting user's access
+ scope, which is a stricter security requirement than the dataframe
+ itself has. Keying them separately from :meth:`query_cache_key` keeps
+ that scoping from forcing every distinct viewer of an annotated chart
+ onto their own full copy of the (potentially much larger) shared
+ dataframe: users with the same access scope share this key too.
"""
- Cache-key material binding cached annotation data to its security
- context.
+ if not query_obj or not query_obj.annotation_layers:
+ return None
+ return self.query_cache_key(
+ query_obj, annotation_context=self._annotation_cache_context(query_obj)
+ )
- Annotation payloads are fetched per requesting user and stored on the
- same cache entry as the dataframe, so the key also binds the requesting
- user and, for chart-backed layers, the RLS clauses of the referenced
- chart's datasource.
+ def _annotation_cache_context(self, query_obj: QueryObject) -> dict[str, Any]:
+ """
+ Cache-key material binding annotation data to its security *scope* so
+ users with the same access share a cache entry and users with a
+ different scope β or no access β never read each other's data.
+
+ * NATIVE layers: the ``can_read`` permission on ``Annotation``, the
+ only user-dependent dimension of these global records.
+ * Chart-backed (``line``/``table``) layers: see
+ :meth:`_annotation_source_scope`.
"""
- source_rls: dict[str, list[str] | None] = {}
+ context: dict[str, Any] = {}
+
+ if any(
+ layer.get("sourceType") == "NATIVE" for layer in query_obj.annotation_layers
+ ):
+ context["annotation_read"] = security_manager.can_access(
+ "can_read", "Annotation"
+ )
+
+ source_scope: dict[str, Any] = {}
for layer in query_obj.annotation_layers:
if layer.get("sourceType") not in ("line", "table"):
continue
layer_value = layer.get("value")
- chart = (
- ChartDAO.find_by_id(layer_value) if layer_value is not None else None
+ source_scope[str(layer_value)] = self._annotation_source_scope(layer_value)
+ if source_scope:
+ context["source_scope"] = source_scope
+
+ return context
+
+ def _annotation_source_scope(self, layer_value: Any) -> dict[str, Any]:
+ """
+ Access and data-identity cache-key material for one chart-backed
+ annotation layer.
+
+ ``access`` keeps a user denied the referenced chart's datasource from
+ reading an authorized user's cached payload. ``data_key`` is the
+ annotation chart's own query cache key(s), which already capture the
+ datasource version, RLS clauses, and any per-user Jinja/virtual-dataset
+ RLS material β reusing it here avoids re-deriving that logic and
+ automatically inherits any future correctness fixes made there.
+ """
+ chart = ChartDAO.find_by_id(layer_value) if layer_value is not None else None
+ datasource = chart.datasource if chart else None
+ if chart is None or datasource is None:
+ return {"access": None, "data_key": None}
+
+ try:
+ access = security_manager.can_access_datasource(datasource)
+ # Fall back to the RLS-clause identity when the chart has no saved
+ # query context to key on.
+ annotation_query_context = chart.get_query_context()
+ data_key: Any = (
+ [
+ annotation_query_context.query_cache_key(query_object)
+ for query_object in annotation_query_context.queries
+ ]
+ if annotation_query_context is not None
+ else security_manager.get_rls_cache_key(datasource)
+ )
+ except SupersetException:
Review Comment:
<div>
<div id="suggestion">
<div id="issue"><b>Narrow except escapes fail-closed</b></div>
<div id="fix">
The try block only catches `SupersetException`, but the derivation can raise others: `chart.get_query_context()` β `QueryContext.query_cache_key` runs `security_manager.get_rls_cache_key` (an RLS DB query) and Jinja `get_extra_cache_keys`, which can raise SQLAlchemy/Jinja errors that are not `SupersetException` subclasses. Such an error escapes the fail-closed fallback and 500s the whole chart-data request. Catch `Exception` (pattern already used at lines 243/579) so derivation failures fall back instead of failing the request.
</div>
</div>
<small><i>Code Review Run #66423b</i></small>
</div>
---
Should Bito avoid suggestions like this for future reviews? (<a href=https://alpha.bito.ai/home/ai-agents/review-rules>Manage Rules</a>)
- [ ] Yes, avoid them
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] fix(charts): decouple annotation cache from the shared dataframe cache [superset]
Posted by "sadpandajoe (via GitHub)" <gi...@apache.org>.
sadpandajoe commented on code in PR #44406:
URL: https://github.com/apache/superset/pull/44406#discussion_r4151108664
##########
superset/common/query_context_processor.py:
##########
@@ -448,31 +479,167 @@ def query_cache_key(self, query_obj: QueryObject, **kwargs: Any) -> str | None:
)
return cache_key
- def _annotation_cache_context(self, query_obj: QueryObject) -> dict[str, Any]:
+ def annotation_cache_key(self, query_obj: QueryObject) -> str | None:
+ """
+ Cache key for this query's annotation-layer payload, or ``None`` when
+ the query has no annotation layers.
+
+ Annotation payloads are fetched under the requesting user's access
+ scope, which is a stricter security requirement than the dataframe
+ itself has. Keying them separately from :meth:`query_cache_key` keeps
+ that scoping from forcing every distinct viewer of an annotated chart
+ onto their own full copy of the (potentially much larger) shared
+ dataframe: users with the same access scope share this key too.
"""
- Cache-key material binding cached annotation data to its security
- context.
+ if not query_obj or not query_obj.annotation_layers:
+ return None
+ return self.query_cache_key(
+ query_obj, annotation_context=self._annotation_cache_context(query_obj)
+ )
- Annotation payloads are fetched per requesting user and stored on the
- same cache entry as the dataframe, so the key also binds the requesting
- user and, for chart-backed layers, the RLS clauses of the referenced
- chart's datasource.
+ def _annotation_cache_context(self, query_obj: QueryObject) -> dict[str, Any]:
+ """
+ Cache-key material binding annotation data to its security *scope* so
+ users with the same access share a cache entry and users with a
+ different scope β or no access β never read each other's data.
+
+ * NATIVE layers: the ``can_read`` permission on ``Annotation``, the
+ only user-dependent dimension of these global records.
+ * Chart-backed (``line``/``table``) layers: see
+ :meth:`_annotation_source_scope`.
"""
- source_rls: dict[str, list[str] | None] = {}
+ context: dict[str, Any] = {}
+
+ if any(
+ layer.get("sourceType") == "NATIVE" for layer in query_obj.annotation_layers
+ ):
+ context["annotation_read"] = security_manager.can_access(
+ "can_read", "Annotation"
+ )
+
+ source_scope: dict[str, Any] = {}
for layer in query_obj.annotation_layers:
if layer.get("sourceType") not in ("line", "table"):
continue
layer_value = layer.get("value")
+ source_scope[str(layer_value)] = self._annotation_source_scope(layer_value)
+ if source_scope:
+ context["source_scope"] = source_scope
+
+ return context
+
+ def _annotation_source_scope(self, layer_value: Any) -> dict[str, Any]:
+ """
+ Access and data-identity cache-key material for one chart-backed
+ annotation layer.
+
+ ``access`` keeps a user denied the referenced chart's datasource from
+ reading an authorized user's cached payload. ``data_key`` is the
+ annotation chart's own query cache key(s), which already capture the
+ datasource version, RLS clauses, and any per-user Jinja/virtual-dataset
+ RLS material β reusing it here avoids re-deriving that logic and
+ automatically inherits any future correctness fixes made there.
+ """
+ datasource = None
+ try:
chart = (
ChartDAO.find_by_id(layer_value) if layer_value is not None else None
)
- annotation_datasource = chart.datasource if chart else None
- source_rls[str(layer.get("value"))] = (
- security_manager.get_rls_cache_key(annotation_datasource)
- if annotation_datasource
- else None
+ # resolved_datasource, not datasource: the latter is pinned to
+ # table-backed datasources and resolves to None for a
+ # semantic-view-backed chart, which would otherwise collapse
+ # every requester onto the same {access: None, data_key: None}
+ # scope below regardless of their actual access.
+ datasource = chart.resolved_datasource if chart else None
+ if chart is None or datasource is None:
+ return {"access": None, "data_key": None}
+
+ access = security_manager.can_access_datasource(datasource)
Review Comment:
With ENABLE_VIEWERS and VIEWER_PROMISCUOUS_MODE, a viewer of the referenced chart and a non-viewer can both have `can_access_datasource(...) == False`, but only the viewer passes the saved query context's authorization; after the viewer warms this entry, the non-viewer receives its rows without that check. Should cache hits validate the referenced query context, or should this scope use the same contextual authorization as the live fetch?
##########
superset/common/query_context_processor.py:
##########
@@ -331,6 +342,26 @@ def get_df_payload_result(
# nonce reads the freshly-cached result instead of recomputing it.
self._mark_force_executed(query_obj, cache_key, cache.result_persisted)
+ # Annotation data is fetched per requesting user (and, for chart-backed
+ # layers, scoped by the referenced chart datasource's RLS), so it is
+ # resolved and cached under its own entry β independent of whether the
+ # (shareable) dataframe above was a hit or a miss β rather than forcing
+ # every viewer of the same chart onto their own full dataframe copy.
+ annotation_data: dict[str, Any] = {}
+ if query_obj and annotation_key and cache.status != QueryStatus.FAILED:
+ try:
+ annotation_data = self._get_annotation_data_cached(
Review Comment:
Two viewers with different annotation RLS scopes can now join the same dataframe-keyed async task, but it only warms the executing viewer's annotation entry; the other viewer's post-success synchronous read-back then runs the annotation query in the HTTP request and can time out even though the task succeeded. Should each subscriber's scoped annotation data be populated asynchronously before its task completion is reported?
##########
superset/common/query_context_processor.py:
##########
@@ -448,31 +479,167 @@ def query_cache_key(self, query_obj: QueryObject, **kwargs: Any) -> str | None:
)
return cache_key
- def _annotation_cache_context(self, query_obj: QueryObject) -> dict[str, Any]:
+ def annotation_cache_key(self, query_obj: QueryObject) -> str | None:
+ """
+ Cache key for this query's annotation-layer payload, or ``None`` when
+ the query has no annotation layers.
+
+ Annotation payloads are fetched under the requesting user's access
+ scope, which is a stricter security requirement than the dataframe
+ itself has. Keying them separately from :meth:`query_cache_key` keeps
+ that scoping from forcing every distinct viewer of an annotated chart
+ onto their own full copy of the (potentially much larger) shared
+ dataframe: users with the same access scope share this key too.
"""
- Cache-key material binding cached annotation data to its security
- context.
+ if not query_obj or not query_obj.annotation_layers:
+ return None
+ return self.query_cache_key(
+ query_obj, annotation_context=self._annotation_cache_context(query_obj)
+ )
- Annotation payloads are fetched per requesting user and stored on the
- same cache entry as the dataframe, so the key also binds the requesting
- user and, for chart-backed layers, the RLS clauses of the referenced
- chart's datasource.
+ def _annotation_cache_context(self, query_obj: QueryObject) -> dict[str, Any]:
+ """
+ Cache-key material binding annotation data to its security *scope* so
+ users with the same access share a cache entry and users with a
+ different scope β or no access β never read each other's data.
+
+ * NATIVE layers: the ``can_read`` permission on ``Annotation``, the
+ only user-dependent dimension of these global records.
+ * Chart-backed (``line``/``table``) layers: see
+ :meth:`_annotation_source_scope`.
"""
- source_rls: dict[str, list[str] | None] = {}
+ context: dict[str, Any] = {}
+
+ if any(
+ layer.get("sourceType") == "NATIVE" for layer in query_obj.annotation_layers
+ ):
+ context["annotation_read"] = security_manager.can_access(
+ "can_read", "Annotation"
+ )
+
+ source_scope: dict[str, Any] = {}
for layer in query_obj.annotation_layers:
if layer.get("sourceType") not in ("line", "table"):
continue
layer_value = layer.get("value")
+ source_scope[str(layer_value)] = self._annotation_source_scope(layer_value)
+ if source_scope:
+ context["source_scope"] = source_scope
+
+ return context
+
+ def _annotation_source_scope(self, layer_value: Any) -> dict[str, Any]:
+ """
+ Access and data-identity cache-key material for one chart-backed
+ annotation layer.
+
+ ``access`` keeps a user denied the referenced chart's datasource from
+ reading an authorized user's cached payload. ``data_key`` is the
+ annotation chart's own query cache key(s), which already capture the
+ datasource version, RLS clauses, and any per-user Jinja/virtual-dataset
+ RLS material β reusing it here avoids re-deriving that logic and
+ automatically inherits any future correctness fixes made there.
+ """
+ datasource = None
+ try:
chart = (
ChartDAO.find_by_id(layer_value) if layer_value is not None else None
)
- annotation_datasource = chart.datasource if chart else None
- source_rls[str(layer.get("value"))] = (
- security_manager.get_rls_cache_key(annotation_datasource)
- if annotation_datasource
- else None
+ # resolved_datasource, not datasource: the latter is pinned to
+ # table-backed datasources and resolves to None for a
+ # semantic-view-backed chart, which would otherwise collapse
+ # every requester onto the same {access: None, data_key: None}
+ # scope below regardless of their actual access.
+ datasource = chart.resolved_datasource if chart else None
+ if chart is None or datasource is None:
+ return {"access": None, "data_key": None}
+
+ access = security_manager.can_access_datasource(datasource)
+ # Fall back to the RLS-clause identity when the chart has no saved
+ # query context to key on.
+ annotation_query_context = chart.get_query_context()
+ data_key: Any = (
+ [
+ annotation_query_context.query_cache_key(query_object)
Review Comment:
A time-grain override can make this shared key omit per-user Jinja material: if the saved monthly query's RLS skips `current_user_id()` but the executed daily override calls it, the next viewer can receive the first viewer's annotation rows. Should the scope be derived from the same overridden query context that `get_viz_annotation_data()` executes?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] fix(charts): decouple annotation cache from the shared dataframe cache [superset]
Posted by "rusackas (via GitHub)" <gi...@apache.org>.
rusackas commented on code in PR #44406:
URL: https://github.com/apache/superset/pull/44406#discussion_r4297984123
##########
superset/common/query_context_processor.py:
##########
@@ -449,34 +481,189 @@ def query_cache_key(self, query_obj: QueryObject, **kwargs: Any) -> str | None:
)
return cache_key
- def _annotation_cache_context(self, query_obj: QueryObject) -> dict[str, Any]:
+ def annotation_cache_key(self, query_obj: QueryObject) -> str | None:
+ """
+ Cache key for this query's annotation-layer payload, or ``None`` when
+ the query has no annotation layers.
+
+ Annotation payloads are fetched under the requesting user's access
+ scope, which is a stricter security requirement than the dataframe
+ itself has. Keying them separately from :meth:`query_cache_key` keeps
+ that scoping from forcing every distinct viewer of an annotated chart
+ onto their own full copy of the (potentially much larger) shared
+ dataframe: users with the same access scope share this key too.
"""
- Cache-key material binding cached annotation data to its security
- context.
+ if not query_obj or not query_obj.annotation_layers:
+ return None
+ return self.query_cache_key(
+ query_obj, annotation_context=self._annotation_cache_context(query_obj)
+ )
- Annotation payloads are fetched per requesting user and stored on the
- same cache entry as the dataframe, so the key also binds the requesting
- user and, for chart-backed layers, the RLS clauses of the referenced
- chart's datasource.
+ def _annotation_cache_context(self, query_obj: QueryObject) -> dict[str, Any]:
+ """
+ Cache-key material binding annotation data to its security *scope* so
+ users with the same access share a cache entry and users with a
+ different scope β or no access β never read each other's data.
+
+ * NATIVE layers: the ``can_read`` permission on ``Annotation``, the
+ only user-dependent dimension of these global records.
+ * Chart-backed (``line``/``table``) layers: see
+ :meth:`_annotation_source_scope`.
"""
- source_rls: dict[str, list[str] | None] = {}
+ context: dict[str, Any] = {}
+
+ if any(
+ layer.get("sourceType") == "NATIVE" for layer in query_obj.annotation_layers
+ ):
+ context["annotation_read"] = security_manager.can_access(
+ "can_read", "Annotation"
+ )
+
+ source_scope: dict[str, Any] = {}
for layer in query_obj.annotation_layers:
if (
layer.get("sourceType")
not in ANNOTATION_SOURCE_TYPES_WITH_CHART_REFERENCE
):
continue
layer_value = layer.get("value")
+ source_scope[str(layer_value)] = self._annotation_source_scope(layer)
+ if source_scope:
+ context["source_scope"] = source_scope
+
+ return context
+
+ def _annotation_source_scope(self, layer: dict[str, Any]) -> dict[str, Any]:
+ """
+ Access and data-identity cache-key material for one chart-backed
+ annotation layer.
+
+ ``access`` keeps a user denied the referenced chart's datasource from
+ reading an authorized user's cached payload. When the chart has a
+ saved query context, this runs the *same* authorization path
+ :meth:`get_viz_annotation_data` executes
+ (``QueryContext.raise_for_access``) rather than the coarser
+ :meth:`SecurityManager.can_access_datasource` β a requester whose
+ access comes from a dashboard/viewer-promiscuous-mode bypass (which
+ depends on the chart's own saved ``form_data``, e.g. its
+ ``slice_id``/``dashboardId``) would otherwise still fail that coarser,
+ context-free check and collapse onto the same denied scope as a
+ genuinely unauthorized requester, letting the latter read the
+ former's cached payload. ``data_key`` is the annotation chart's own
+ query cache key(s) β derived from the same, override-applied query
+ objects actually executed (see :meth:`_apply_annotation_overrides`),
+ so it captures the datasource version, RLS clauses, and any per-user
+ Jinja/virtual-dataset RLS material exactly as the live fetch would,
+ including material an override only introduces at a finer grain.
+ Reusing this logic avoids re-deriving it and automatically inherits
+ any future correctness fixes made there.
+ """
+ layer_value = layer.get("value")
+ datasource = None
+ try:
chart = (
ChartDAO.find_by_id(layer_value) if layer_value is not None else None
)
- annotation_datasource = chart.datasource if chart else None
- source_rls[str(layer.get("value"))] = (
- security_manager.get_rls_cache_key(annotation_datasource)
- if annotation_datasource
- else None
+ # resolved_datasource, not datasource: the latter is pinned to
+ # table-backed datasources and resolves to None for a
+ # semantic-view-backed chart, which would otherwise collapse
+ # every requester onto the same {access: None, data_key: None}
+ # scope below regardless of their actual access.
+ datasource = chart.resolved_datasource if chart else None
+ if chart is None or datasource is None:
+ return {"access": None, "data_key": None}
+
+ annotation_query_context = chart.get_query_context()
+ if annotation_query_context is not None:
+ self._apply_annotation_overrides(annotation_query_context, layer)
+ try:
+ annotation_query_context.raise_for_access()
+ access: Any = True
+ except SupersetSecurityException:
+ access = False
+ data_key: Any = [
+ annotation_query_context.query_cache_key(query_object)
Review Comment:
That detector is gone now. The key is derived from the override-applied query objects the live fetch actually runs, through `query_cache_key` / `get_extra_cache_keys`, which renders the Jinja itself, so spacing like `current_user_id ()` is handled by the template engine rather than a syntactic check.
##########
superset/common/query_context_processor.py:
##########
@@ -332,6 +344,30 @@ def get_df_payload_result(
# nonce reads the freshly-cached result instead of recomputing it.
self._mark_force_executed(query_obj, cache_key, cache.result_persisted)
+ # Annotation data is fetched per requesting user (and, for chart-backed
+ # layers, scoped by the referenced chart datasource's RLS), so it is
+ # resolved and cached under its own entry β independent of whether the
+ # (shareable) dataframe above was a hit or a miss β rather than forcing
+ # every viewer of the same chart onto their own full dataframe copy.
+ annotation_data: dict[str, Any] = {}
+ if (
+ query_obj
+ and query_obj.annotation_layers
+ and cache.status != QueryStatus.FAILED
+ ):
+ try:
+ annotation_data = self._get_annotation_data_cached(
Review Comment:
Good catch. The annotation fetch is now timed and summed into `data_acquisition_ns` (so a dataframe cache hit with annotation SQL no longer reports null). Added a test for it.
##########
tests/unit_tests/common/test_query_context_processor.py:
##########
@@ -117,25 +122,350 @@ def processor(mock_query_context):
return processor
-def test_query_cache_key_binds_annotation_data_to_requesting_user(processor):
- """The cache key for annotated queries must differ per requesting user."""
+def test_annotation_cache_key_binds_native_annotation_read_scope(processor) -> None:
+ """The annotation cache key for NATIVE layers must differ when the
+ requester's ``can_read`` (Annotation) access differs -- not who they are."""
query_obj = MagicMock()
query_obj.annotation_layers = [{"sourceType": "NATIVE", "name": "a", "value": 1}]
- with (
- patch(
- "superset.common.query_context_processor.get_user_id",
- side_effect=[1, 2],
- ),
- patch("superset.common.query_context_processor.security_manager"),
- ):
- processor.query_cache_key(query_obj)
- processor.query_cache_key(query_obj)
+ # ``security_manager`` autodetects as an async spec under a bare
+ # ``patch()`` (its real object trips ``unittest.mock``'s coroutine
+ # inference), which would silently turn every attribute access into an
+ # ``AsyncMock`` returning a fresh unawaited coroutine per call -- always
+ # unequal to itself and never equal to a configured return value. Forcing
+ # ``new_callable=MagicMock`` keeps these synchronous, as the real object
+ # is.
+ with patch(
+ "superset.common.query_context_processor.security_manager",
+ new_callable=MagicMock,
+ ) as security_manager:
+ security_manager.can_access.side_effect = [True, False]
+ processor.annotation_cache_key(query_obj)
+ processor.annotation_cache_key(query_obj)
contexts = [
call.kwargs["annotation_context"] for call in query_obj.cache_key.call_args_list
]
assert contexts[0] != contexts[1]
+def test_annotation_cache_key_shares_across_same_access_scope() -> None:
+ """Two distinct requesters (separate processor/query-object instances,
+ standing in for two different requests) with identical access scope must
+ produce identical annotation-context material. Reusing a single
+ processor/query_obj across both calls (as this test previously did)
+ would pass trivially regardless of whether the key is scope-based or
+ identity-based, since nothing about "who's asking" would ever vary."""
+ layer = {"sourceType": "NATIVE", "name": "a", "value": 1}
+ processor_a = QueryContextProcessor(MagicMock())
+ processor_b = QueryContextProcessor(MagicMock())
+ query_obj_a = MagicMock(annotation_layers=[layer])
+ query_obj_b = MagicMock(annotation_layers=[layer])
+
+ with patch(
+ "superset.common.query_context_processor.security_manager",
+ new_callable=MagicMock,
+ ) as security_manager:
+ security_manager.can_access.return_value = True
+ context_a = processor_a._annotation_cache_context(query_obj_a)
+ context_b = processor_b._annotation_cache_context(query_obj_b)
+
+ assert context_a == context_b
+
+
+def test_query_cache_key_does_not_bind_annotation_scope(processor) -> None:
+ """The dataframe cache key must stay shared across viewers of the same
+ chart, even when the query has annotation layers β only the separate
+ annotation cache key (see above) carries access-scope material."""
+ query_obj = MagicMock()
+ query_obj.annotation_layers = [{"sourceType": "NATIVE", "name": "a", "value": 1}]
+ with patch(
+ "superset.common.query_context_processor.security_manager",
+ new_callable=MagicMock,
+ ):
+ processor.query_cache_key(query_obj)
+ processor.query_cache_key(query_obj)
+ for call in query_obj.cache_key.call_args_list:
+ assert "annotation_context" not in call.kwargs
+
+
+@pytest.fixture
+def mock_annotation_chart() -> Iterator[MagicMock]:
+ """A found chart, wired as the referenced chart for
+ ``_annotation_source_scope`` tests -- factors out the repeated
+ ``ChartDAO.find_by_id`` patch those tests all need."""
+ chart = MagicMock()
+ with patch(
+ "superset.common.query_context_processor.ChartDAO.find_by_id",
+ return_value=chart,
+ ):
+ yield chart
+
+
+def test_annotation_source_scope_binds_datasource_access(
+ processor: QueryContextProcessor, mock_annotation_chart: MagicMock
+) -> None:
+ """A chart-backed annotation layer's scope must differ when the
+ requester's access to the referenced datasource differs."""
+ mock_annotation_chart.get_query_context.return_value = None
+ with patch(
+ "superset.common.query_context_processor.security_manager",
+ new_callable=MagicMock,
+ ) as security_manager:
+ security_manager.can_access_datasource.side_effect = [True, False]
+ security_manager.get_rls_cache_key.return_value = []
+ scope_a = processor._annotation_source_scope({"value": 1})
+ scope_b = processor._annotation_source_scope({"value": 1})
+ assert scope_a is not None
+ assert scope_b is not None
+ assert scope_a != scope_b
+ assert scope_a["access"] is True
+ assert scope_b["access"] is False
+
+
+def test_annotation_source_scope_reuses_referenced_chart_cache_key(
+ processor: QueryContextProcessor, mock_annotation_chart: MagicMock
+) -> None:
+ """When the referenced chart has a saved query context, its own cache
+ key(s) -- covering RLS and per-user Jinja/virtual-dataset material -- are
+ reused rather than re-derived."""
+ mock_query_object: MagicMock = MagicMock()
+ mock_query_context: MagicMock = MagicMock()
+ mock_query_context.queries = [mock_query_object]
+ mock_query_context.query_cache_key.return_value = "referenced-chart-key"
+ mock_annotation_chart.get_query_context.return_value = mock_query_context
+ with patch(
+ "superset.common.query_context_processor.security_manager",
+ new_callable=MagicMock,
+ ) as security_manager:
+ security_manager.can_access_datasource.return_value = True
+ scope = processor._annotation_source_scope({"value": 1})
+ assert scope == {"access": True, "data_key": ["referenced-chart-key"]}
+ mock_query_context.query_cache_key.assert_called_once_with(mock_query_object)
+
+
+def test_annotation_source_scope_uses_live_fetch_authorization(
+ processor: QueryContextProcessor, mock_annotation_chart: MagicMock
+) -> None:
+ """When the referenced chart has a saved query context, ``access`` must
+ come from that context's own ``raise_for_access`` -- the same
+ authorization path the live fetch in ``get_viz_annotation_data`` uses --
+ not the coarser, context-free ``can_access_datasource``. A requester
+ denied by ``can_access_datasource`` but granted via a bypass that depends
+ on the chart's own saved form_data (e.g. a dashboard/viewer-promiscuous
+ bypass) must get a scope distinct from one truly denied, or the latter
+ could read the former's cached payload."""
+ mock_query_object: MagicMock = MagicMock()
+ mock_query_context: MagicMock = MagicMock()
+ mock_query_context.queries = [mock_query_object]
+ mock_query_context.query_cache_key.return_value = "referenced-chart-key"
+ mock_annotation_chart.get_query_context.return_value = mock_query_context
+ with patch(
+ "superset.common.query_context_processor.security_manager",
+ new_callable=MagicMock,
+ ) as security_manager:
+ # Coarse proxy says "denied" for both -- the real authorization
+ # path must be what actually decides access.
+ security_manager.can_access_datasource.return_value = False
+ mock_query_context.raise_for_access.side_effect = [
+ None,
+ SupersetSecurityException(MagicMock()),
+ ]
+ scope_a = processor._annotation_source_scope({"value": 1})
+ scope_b = processor._annotation_source_scope({"value": 1})
+ assert scope_a is not None
+ assert scope_b is not None
+ assert scope_a["access"] is True
+ assert scope_b["access"] is False
+ assert scope_a != scope_b
+ security_manager.can_access_datasource.assert_not_called()
+
+
+def test_annotation_source_scope_applies_overrides_before_keying(
+ processor: QueryContextProcessor, mock_annotation_chart: MagicMock
+) -> None:
+ """A time-grain/time-range override on the annotation layer must be
+ applied to the referenced chart's query objects *before* deriving the
+ cache key, mirroring ``get_viz_annotation_data`` exactly -- otherwise the
+ key can omit per-user Jinja/RLS material an override only introduces at a
+ finer grain."""
+ mock_query_object = MagicMock()
+ mock_query_object.extras = {}
+ mock_query_context = MagicMock()
+ mock_query_context.queries = [mock_query_object]
+ mock_query_context.query_cache_key.return_value = "referenced-chart-key"
+ mock_annotation_chart.get_query_context.return_value = mock_query_context
+ layer = {
+ "value": 1,
+ "overrides": {"time_grain_sqla": "P1D", "time_range": "Last week"},
+ }
+ with patch(
+ "superset.common.query_context_processor.security_manager",
+ new_callable=MagicMock,
+ ):
+ processor._annotation_source_scope(layer)
Review Comment:
Makes sense. `query_cache_key` now has a `side_effect` that asserts the overridden grain, time range and bounds are already set when it's called, so reordering would fail the test.
##########
tests/unit_tests/common/test_query_context_processor.py:
##########
@@ -2238,6 +2568,252 @@ def test_get_df_payload_no_warning_when_not_memory_limited() -> None:
assert result["warning"] is None
+def test_get_df_payload_result_decouples_annotation_cache_from_dataframe_cache() -> (
+ None
+):
+ """
+ The dataframe cache entry must stay shareable across viewers, and
+ annotation-layer data must be resolved through its own (per-user) cache
+ path -- not stored on the dataframe's cache entry -- so that two viewers
+ of the same annotated chart share one dataframe cache hit while each
+ still gets their own annotation-security-scoped payload.
+ """
+ from superset.common.query_object import QueryObject
+
+ mock_query_context = MagicMock()
+ mock_query_context.force = False
+ mock_datasource = MagicMock()
+ mock_datasource.column_names = ["col1"]
+
+ processor = QueryContextProcessor(mock_query_context)
+ processor._qc_datasource = mock_datasource
+
+ query_obj = QueryObject(
+ datasource=mock_datasource,
+ columns=["col1"],
+ annotation_layers=[
+ {
+ "annotationType": "EVENT",
+ "sourceType": "NATIVE",
+ "name": "a",
+ "value": 1,
+ }
+ ],
+ )
+
+ class MockCache:
+ def __init__(self):
+ self.is_loaded = True
+ self.applied_filter_columns = ["col1"]
+ self.df = pd.DataFrame({"col1": [1, 2, 3]})
+ self.query = ""
+ self.status = "success"
+ self.cache_dttm = "2024-01-01T00:00:00"
+ self.queried_dttm = "2024-01-01T00:00:00"
+ self.stacktrace = None
+ self.error_message = None
+ self.is_cached = True
+ self.sql_rowcount = 0
+ self.cache_value = None
+ self.applied_template_filters = []
+ self.rejected_filter_columns = []
+ self.annotation_data = {"stale": "should not be used"}
+ self.bq_memory_limited = False
+ self.bq_memory_limited_row_count = 0
+ self.result_persisted = False
+ self.set_query_result = MagicMock()
+
+ mock_cache = MockCache()
+
+ with (
+ patch(
+ "superset.common.query_context_processor.QueryCacheManager"
+ ) as mock_cache_manager,
+ patch.object(query_obj, "validate", return_value=None),
+ patch.object(processor, "query_cache_key", return_value="df-key"),
+ patch.object(processor, "annotation_cache_key", return_value="ann-key"),
+ patch.object(
+ processor, "_get_annotation_data_cached", return_value={"a": [1, 2]}
+ ) as mock_get_annotation,
+ patch.object(processor, "get_cache_timeout", return_value=3600),
+ ):
+ mock_cache_manager.get.return_value = mock_cache
+ result = processor.get_df_payload(query_obj, force_cached=False)
+
+ # The dataframe cache is a hit, so the (expensive) query is never re-run,
+ # and its cache entry is never rewritten.
+ mock_cache.set_query_result.assert_not_called()
+
+ # Annotation data is resolved through the separate, per-user cache path,
+ # keyed by the dedicated annotation cache key -- not the dataframe's key.
+ mock_get_annotation.assert_called_once()
+ _, kwargs = mock_get_annotation.call_args
+ assert kwargs["cache_key"] == "ann-key"
Review Comment:
Good point. The decouple test now asserts `force_cached` and `force_query` are forwarded at the call site, and I added a case that runs the real helper with `force_cached=True` and no annotation entry, expecting `CacheLoadError` and no live fetch.
##########
superset/common/query_context_processor.py:
##########
@@ -72,7 +74,6 @@
get_column_name,
get_column_names_from_columns,
get_column_names_from_metrics,
- get_user_id,
is_adhoc_column,
Review Comment:
Those imports are already gone in the current head, and ruff (F401) passes on the file.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] fix(charts): decouple annotation cache from the shared dataframe cache [superset]
Posted by "rusackas (via GitHub)" <gi...@apache.org>.
rusackas commented on code in PR #44406:
URL: https://github.com/apache/superset/pull/44406#discussion_r4297986329
##########
superset/common/query_context_processor.py:
##########
@@ -343,6 +355,32 @@ def get_df_payload_result(
# nonce reads the freshly-cached result instead of recomputing it.
self._mark_force_executed(query_obj, cache_key, cache.result_persisted)
+ # Annotation data is fetched per requesting user (and, for chart-backed
+ # layers, scoped by the referenced chart datasource's RLS), so it is
+ # resolved and cached under its own entry β independent of whether the
+ # (shareable) dataframe above was a hit or a miss β rather than forcing
+ # every viewer of the same chart onto their own full dataframe copy.
+ annotation_data: dict[str, Any] = {}
+ if (
+ query_obj
+ and query_obj.annotation_layers
+ and cache.status != QueryStatus.FAILED
Review Comment:
Added tests driving `get_df_payload_result` for a failed dataframe (annotation helper not called, `annotation_data == {}`, original error kept) and for an annotation error on a good dataframe. The persisted-only marker write is covered at the helper boundary in a separate test that parametrizes `set_and_log_cache` returning True/False.
##########
superset/common/query_context_processor.py:
##########
@@ -937,6 +1119,51 @@ def get_native_annotation_data(query_obj: QueryObject) -> dict[str, Any]:
annotation_data[layer_name] = result
return annotation_data
+ @staticmethod
+ def _apply_annotation_overrides(
+ query_context: QueryContext, annotation_layer: dict[str, Any]
+ ) -> None:
+ """
+ Apply a chart-backed annotation layer's per-request overrides (time
+ grain, time range) to its saved query context's queries, in place.
+
+ Shared by :meth:`get_viz_annotation_data` (which executes the
+ overridden query) and :meth:`_annotation_source_scope` (which derives
+ its cache key from it), so the key always reflects the query that
+ actually runs: an override can introduce per-user Jinja/RLS material
+ the saved, un-overridden query lacks (e.g. a template that only calls
+ ``current_user_id()`` at a finer time grain), which deriving the key
+ from the un-overridden query would silently miss.
+ """
+ # Only the source chart's rows are used. Dropping its own annotation
+ # layers also stops charts that annotate each other (A -> B -> A)
+ # from recursing.
+ for query_object in query_context.queries:
+ query_object.annotation_layers = []
+
+ if not (overrides := annotation_layer.get("overrides")):
+ return
+
+ if time_grain_sqla := overrides.get("time_grain_sqla"):
+ for query_object in query_context.queries:
+ query_object.extras["time_grain_sqla"] = time_grain_sqla
+
+ if time_range := overrides.get("time_range"):
+ from_dttm, to_dttm = get_since_until_from_time_range(time_range)
+
+ for query_object in query_context.queries:
+ query_object.from_dttm = from_dttm
+ query_object.to_dttm = to_dttm
+ # Keep the logical range alongside the resolved bounds:
+ # ``QueryObject.cache_key`` keys on ``time_range`` when set
+ # (dropping the bounds), so a relative override such as
+ # "Last week" keeps one key as time passes instead of
+ # contributing freshly resolved timestamps on every request
+ # (which would defeat cache and force-nonce reuse), and an
+ # override on a saved query without a range of its own is
+ # still told apart from a different override.
+ query_object.time_range = time_range
Review Comment:
Agreed, that was a real mismatch. The override range is now resolved per query with the saved query's `time_shift` (same call the factory makes), so the window that runs matches what a normal run of that range and shift would run. Added a test.
##########
tests/unit_tests/semantic_layers/simple_cache_clear_test.py:
##########
@@ -307,32 +317,24 @@ def test_annotation_result_key_uses_saved_source_generation(
from superset.common.query_context_processor import QueryContextProcessor
view: SemanticView = SemanticView(name="view", semantic_layer=layer)
- chart: Mock = Mock(query_context='{"datasource":{"id":17,"type":"semantic_view"}}')
+ chart: Mock = _saved_context_chart(view)
query: Mock = Mock(annotation_layers=[{"sourceType": "line", "value": 7}])
processor: QueryContextProcessor = QueryContextProcessor(Mock())
- get_datasource: Mock
with (
app.app_context(),
patch(
"superset.common.query_context_processor.ChartDAO.find_by_id",
return_value=chart,
),
- patch(
- "superset.daos.datasource.DatasourceDAO.get_datasource",
- return_value=view,
- ) as get_datasource,
- patch(
- "superset.common.query_context_processor.security_manager.get_rls_cache_key",
- return_value=[],
- ),
):
- first: dict[str, Any] = processor._annotation_cache_context(query)
+ first: dict[str, Any] | None = processor._annotation_cache_context(query)
+ assert first is not None
layer.cache_version = 1
session.info.clear()
view.forget_metadata()
assert processor._annotation_cache_context(query) != first
Review Comment:
Fair. Both tests now require the rotated context to be non-None and compare the version field (changed) against the rest of the scope (unchanged).
##########
tests/unit_tests/common/test_query_context_processor.py:
##########
@@ -126,25 +131,350 @@ def processor(mock_query_context):
return processor
-def test_query_cache_key_binds_annotation_data_to_requesting_user(processor):
- """The cache key for annotated queries must differ per requesting user."""
+def test_annotation_cache_key_binds_native_annotation_read_scope(processor) -> None:
+ """The annotation cache key for NATIVE layers must differ when the
+ requester's ``can_read`` (Annotation) access differs -- not who they are."""
query_obj = MagicMock()
query_obj.annotation_layers = [{"sourceType": "NATIVE", "name": "a", "value": 1}]
- with (
- patch(
- "superset.common.query_context_processor.get_user_id",
- side_effect=[1, 2],
- ),
- patch("superset.common.query_context_processor.security_manager"),
- ):
- processor.query_cache_key(query_obj)
- processor.query_cache_key(query_obj)
+ # ``security_manager`` autodetects as an async spec under a bare
+ # ``patch()`` (its real object trips ``unittest.mock``'s coroutine
+ # inference), which would silently turn every attribute access into an
+ # ``AsyncMock`` returning a fresh unawaited coroutine per call -- always
+ # unequal to itself and never equal to a configured return value. Forcing
+ # ``new_callable=MagicMock`` keeps these synchronous, as the real object
+ # is.
+ with patch(
+ "superset.common.query_context_processor.security_manager",
+ new_callable=MagicMock,
+ ) as security_manager:
+ security_manager.can_access.side_effect = [True, False]
+ processor.annotation_cache_key(query_obj)
+ processor.annotation_cache_key(query_obj)
contexts = [
call.kwargs["annotation_context"] for call in query_obj.cache_key.call_args_list
]
assert contexts[0] != contexts[1]
+def test_annotation_cache_key_shares_across_same_access_scope() -> None:
+ """Two distinct requesters (separate processor/query-object instances,
+ standing in for two different requests) with identical access scope must
+ produce identical annotation-context material. Reusing a single
+ processor/query_obj across both calls (as this test previously did)
+ would pass trivially regardless of whether the key is scope-based or
+ identity-based, since nothing about "who's asking" would ever vary."""
+ layer = {"sourceType": "NATIVE", "name": "a", "value": 1}
+ processor_a = QueryContextProcessor(MagicMock())
+ processor_b = QueryContextProcessor(MagicMock())
+ query_obj_a = MagicMock(annotation_layers=[layer])
+ query_obj_b = MagicMock(annotation_layers=[layer])
+
+ with patch(
+ "superset.common.query_context_processor.security_manager",
+ new_callable=MagicMock,
+ ) as security_manager:
+ security_manager.can_access.return_value = True
+ context_a = processor_a._annotation_cache_context(query_obj_a)
+ context_b = processor_b._annotation_cache_context(query_obj_b)
+
+ assert context_a == context_b
Review Comment:
Good call. The test now patches `get_user_id` to return different values per call (both on the processor module and `superset.utils.core`) and still expects equal contexts.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] fix(charts): decouple annotation cache from the shared dataframe cache [superset]
Posted by "sadpandajoe (via GitHub)" <gi...@apache.org>.
sadpandajoe commented on code in PR #44406:
URL: https://github.com/apache/superset/pull/44406#discussion_r4277065243
##########
tests/unit_tests/common/test_query_context_processor.py:
##########
@@ -2639,6 +3215,101 @@ def test_mark_force_executed_noop_without_nonce(processor, mock_query_context):
cache_manager.data_cache.set.assert_not_called()
+# =============================================================================
+# Annotation-data cache decoupled from the dataframe cache
+# =============================================================================
+
+
+def test_get_annotation_data_cached_reads_from_cache(
+ processor: QueryContextProcessor,
+) -> None:
+ """A hit on the annotation-specific key skips recomputation entirely."""
+ with patch(
+ "superset.common.query_context_processor.cache_manager"
+ ) as cache_manager:
+ cache_manager.data_cache.get.return_value = {"annotation_data": {"a": 1}}
+ with patch.object(processor, "get_annotation_data") as mock_get:
+ result = processor._get_annotation_data_cached(
+ query_obj=MagicMock(),
+ cache_key="ak",
+ force_query=False,
+ force_cached=False,
+ timeout=60,
+ datasource_uid="ds",
+ )
+ assert result == {"a": 1}
+ mock_get.assert_not_called()
+
+
+def test_get_annotation_data_cached_computes_and_caches_on_miss(
+ processor: QueryContextProcessor,
+) -> None:
+ """A miss computes the annotation data and stores it under the scoped key."""
+ with (
+ patch("superset.common.query_context_processor.cache_manager") as cache_manager,
+ patch("superset.common.query_context_processor.set_and_log_cache") as mock_set,
+ ):
+ cache_manager.data_cache.get.return_value = None
+ with patch.object(
+ processor, "get_annotation_data", return_value={"a": 1}
+ ) as mock_get:
+ result = processor._get_annotation_data_cached(
+ query_obj=MagicMock(),
+ cache_key="ak",
+ force_query=False,
+ force_cached=False,
+ timeout=60,
+ datasource_uid="ds",
+ )
+ assert result == {"a": 1}
+ mock_get.assert_called_once()
+ mock_set.assert_called_once_with(
+ cache_manager.data_cache, "ak", {"annotation_data": {"a": 1}}, 60, "ds"
+ )
+
+
+def test_get_annotation_data_cached_force_cached_raises_on_miss(
Review Comment:
The new `except Exception` around `cache_manager.data_cache.get(cache_key)` in `_get_annotation_data_cached` has no test: the only read-error test (`test_resolve_forced_query_forces_when_marker_read_errors`) exercises the marker lookup in `_resolve_forced_query`, not this catch. If that catch were dropped, a cache backend `ConnectionError` would propagate and fail the whole chart-data request, and every test here would still pass. Could you add a case where a dataframe cache hit is paired with `data_cache.get` raising, asserting live annotations are returned without `force_cached` and `CacheLoadError` is raised (no live fetch) with `force_cached=True`?
##########
superset/common/query_context_processor.py:
##########
@@ -474,73 +503,226 @@ def query_cache_key(self, query_obj: QueryObject, **kwargs: Any) -> str | None:
)
return cache_key
- def _annotation_cache_context(self, query_obj: QueryObject) -> dict[str, Any]:
+ def annotation_cache_key(self, query_obj: QueryObject) -> str | None:
"""
- Cache-key material binding cached annotation data to its security
- context.
-
- Annotation payloads are fetched per requesting user and stored on the
- same cache entry as the dataframe, so the key also binds the requesting
- user and, for chart-backed layers, the RLS clauses of the referenced
- chart's datasource.
+ Cache key for this query's annotation-layer payload, or ``None`` when
+ the query has no annotation layers or when some layer's access scope
+ could not be resolved (the payload is then never cached, see
+ :meth:`_annotation_source_scope`).
+
+ Annotation payloads are fetched under the requesting user's access
+ scope, which is a stricter security requirement than the dataframe
+ itself has. Keying them separately from :meth:`query_cache_key` keeps
+ that scoping from forcing every distinct viewer of an annotated chart
+ onto their own full copy of the (potentially much larger) shared
+ dataframe: users with the same access scope share this key too.
"""
- source_rls: dict[str, list[str] | None] = {}
- source_versions: dict[str, tuple[str, str]] = {}
- metadata_versions: dict[str, str] = {}
+ if not query_obj or not query_obj.annotation_layers:
+ return None
+ if (annotation_context := self._annotation_cache_context(query_obj)) is None:
+ return None
+ return self.query_cache_key(query_obj, annotation_context=annotation_context)
+
+ def _annotation_cache_context(
+ self, query_obj: QueryObject
+ ) -> dict[str, Any] | None:
+ """
+ Cache-key material binding annotation data to its security *scope* so
+ users with the same access share a cache entry and users with a
+ different scope β or no access β never read each other's data.
+
+ * NATIVE layers: the ``can_read`` permission on ``Annotation``, the
+ only user-dependent dimension of these global records.
+ * Chart-backed (``line``/``table``) layers: see
+ :meth:`_annotation_source_scope`, recorded per *layer* (not per
+ referenced chart) since two layers can reference the same chart
+ with different overrides, and so different scopes.
+
+ Returns ``None`` when any layer's scope cannot be resolved, meaning
+ the payload must not be cached at all.
+ """
+ context: dict[str, Any] = {}
+
+ if any(
+ layer.get("sourceType") == "NATIVE" for layer in query_obj.annotation_layers
+ ):
+ context["annotation_read"] = security_manager.can_access(
+ "can_read", "Annotation"
+ )
+
+ source_scope: list[dict[str, Any]] = []
for layer in query_obj.annotation_layers:
if (
layer.get("sourceType")
not in ANNOTATION_SOURCE_TYPES_WITH_CHART_REFERENCE
):
continue
- layer_value = layer.get("value")
+ if (scope := self._annotation_source_scope(layer)) is None:
+ return None
+ source_scope.append({"value": str(layer.get("value")), **scope})
+ if source_scope:
+ context["source_scope"] = source_scope
+
+ return context
+
+ def _annotation_source_scope(self, layer: dict[str, Any]) -> dict[str, Any] | None:
+ """
+ Access and data-identity cache-key material for one chart-backed
+ annotation layer, or ``None`` when it cannot be resolved.
+
+ ``None`` means "do not cache": a missing chart, a chart with neither
+ a saved query context nor a resolvable datasource, or any error while
+ deriving the scope would otherwise collapse an authorized and a
+ denied requester onto one shared identity, letting the latter read
+ the former's warmed payload. The caller fetches live instead.
+
+ ``access`` keeps a user denied the referenced chart's datasource from
+ reading an authorized user's cached payload. When the chart has a
+ saved query context, this runs the *same* authorization path
+ :meth:`get_viz_annotation_data` executes
+ (``QueryContext.raise_for_access``) rather than the coarser
+ :meth:`SecurityManager.can_access_datasource` β a requester whose
+ access comes from a dashboard/viewer-promiscuous-mode bypass (which
+ depends on the chart's own saved ``form_data``, e.g. its
+ ``slice_id``/``dashboardId``) would otherwise still fail that coarser,
+ context-free check and collapse onto the same denied scope as a
+ genuinely unauthorized requester, letting the latter read the
+ former's cached payload. ``data_key`` is the annotation chart's own
+ query cache key(s) β derived from the same, override-applied query
+ objects actually executed (see :meth:`_apply_annotation_overrides`),
+ so it captures the datasource version, RLS clauses, and any per-user
+ Jinja/virtual-dataset RLS material exactly as the live fetch would,
+ including material an override only introduces at a finer grain.
+ Reusing this logic avoids re-deriving it and automatically inherits
+ any future correctness fixes made there.
+ """
+ layer_value = layer.get("value")
+ try:
chart = (
ChartDAO.find_by_id(layer_value) if layer_value is not None else None
)
- annotation_datasource: Datasource | None = (
- chart.resolved_datasource if chart else None
- )
- if chart and isinstance(chart.query_context, str):
+ if chart is None:
+ return None
+
+ # The saved query context is what the live fetch executes and
+ # authorizes against, so it is consulted first, independent of
+ # the chart's own datasource pointer: a chart can have its
+ # ``datasource_id`` cleared while its saved query context still
+ # targets a valid dataset, and the live fetch would still work.
+ annotation_query_context = chart.get_query_context()
+ if annotation_query_context is not None:
+ self._apply_annotation_overrides(annotation_query_context, layer)
try:
- source: dict[str, Any] = json.loads(chart.query_context)[
- "datasource"
- ]
- annotation_datasource = DatasourceDAO.get_datasource(
- source["type"], source["id"]
- )
- except (
- ValueError,
- TypeError,
- KeyError,
- DatasourceNotFound,
- DatasourceTypeNotSupportedError,
- DatasourceValueIsIncorrect,
- ):
- annotation_datasource = None
+ annotation_query_context.raise_for_access()
+ access: Any = True
+ except SupersetSecurityException:
+ access = False
+ data_key: Any = [
Review Comment:
When `raise_for_access()` just denied this requester, `access` is `False`, but execution still falls through to `annotation_query_context.query_cache_key(...)`. That call runs `get_extra_cache_keys`, which can compile the referenced dataset's SQL and render its Jinja (including DB-backed macros like `latest_partition`) for a user who has no access to it, on every load of the annotated chart β and nothing is ever cached under an `access: False` scope, so the work repeats. Should the denied branch skip `data_key` (e.g. return `{"access": False}`), since a denied scope never needs a data identity?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] fix(charts): decouple annotation cache from the shared dataframe cache [superset]
Posted by "sadpandajoe (via GitHub)" <gi...@apache.org>.
sadpandajoe commented on code in PR #44406:
URL: https://github.com/apache/superset/pull/44406#discussion_r4302650630
##########
superset/common/query_context_processor.py:
##########
@@ -356,6 +361,39 @@ def get_df_payload_result(
# nonce reads the freshly-cached result instead of recomputing it.
self._mark_force_executed(query_obj, cache_key, cache.result_persisted)
+ # Annotation data is fetched per requesting user (and, for chart-backed
+ # layers, scoped by the referenced chart datasource's RLS), so it is
+ # resolved and cached under its own entry β independent of whether the
+ # (shareable) dataframe above was a hit or a miss β rather than forcing
+ # every viewer of the same chart onto their own full dataframe copy.
+ annotation_data: dict[str, Any] = {}
+ if (
+ query_obj
+ and query_obj.annotation_layers
+ and cache.status != QueryStatus.FAILED
+ ):
+ annotation_start_ns = time.perf_counter_ns()
+ try:
+ annotation_data = self._get_annotation_data_cached(
+ query_obj=query_obj,
+ cache_key=annotation_key,
+ force_query=annotation_force_query,
+ force_cached=force_cached,
+ timeout=self.get_cache_timeout(),
+ datasource_uid=self._qc_datasource.uid,
+ )
+ except SemanticResultCompletenessError:
+ raise
+ except QueryObjectValidationError as ex:
+ cache.error_message = str(ex)
Review Comment:
When an annotation layer fails validation (for example a native layer that was deleted since the chart was saved), this stores a plain `str(ex)` instead of `QueryObjectValidationMessage(ex)`, as the dataframe path does a few lines above. `ChartDataCommand._raise_for_query_error` only raises the validation error type for the latter, so the same failure now surfaces as `ChartDataQueryFailedError`, where it used to be a validation error (400). Should this record `QueryObjectValidationMessage(ex)` so the error type is preserved?
```suggestion
cache.error_message = QueryObjectValidationMessage(ex)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] fix(charts): decouple annotation cache from the shared dataframe cache [superset]
Posted by "rusackas (via GitHub)" <gi...@apache.org>.
rusackas commented on code in PR #44406:
URL: https://github.com/apache/superset/pull/44406#discussion_r4316316304
##########
superset/common/query_context_processor.py:
##########
@@ -331,6 +342,26 @@ def get_df_payload_result(
# nonce reads the freshly-cached result instead of recomputing it.
self._mark_force_executed(query_obj, cache_key, cache.result_persisted)
+ # Annotation data is fetched per requesting user (and, for chart-backed
+ # layers, scoped by the referenced chart datasource's RLS), so it is
+ # resolved and cached under its own entry β independent of whether the
+ # (shareable) dataframe above was a hit or a miss β rather than forcing
+ # every viewer of the same chart onto their own full dataframe copy.
+ annotation_data: dict[str, Any] = {}
+ if query_obj and annotation_key and cache.status != QueryStatus.FAILED:
+ try:
+ annotation_data = self._get_annotation_data_cached(
Review Comment:
I don't think the read-back runs the annotation query in the request. With `force_cached=True` an annotation cache miss for B's scope raises `CacheLoadError`, `_run_async` catches it and submits a fresh task under B, which warms B's scoped entry. The cost is one extra task round trip for B after joining A's task, not an HTTP-request timeout. The only live-in-request case is the unresolved-scope fallback, which is deliberate. Happy to fold the annotation key into the task key if you think the extra round trip matters, but I'd rather not in this PR.
##########
superset/common/query_context_processor.py:
##########
@@ -449,34 +481,189 @@ def query_cache_key(self, query_obj: QueryObject, **kwargs: Any) -> str | None:
)
return cache_key
- def _annotation_cache_context(self, query_obj: QueryObject) -> dict[str, Any]:
+ def annotation_cache_key(self, query_obj: QueryObject) -> str | None:
+ """
+ Cache key for this query's annotation-layer payload, or ``None`` when
+ the query has no annotation layers.
+
+ Annotation payloads are fetched under the requesting user's access
+ scope, which is a stricter security requirement than the dataframe
+ itself has. Keying them separately from :meth:`query_cache_key` keeps
+ that scoping from forcing every distinct viewer of an annotated chart
+ onto their own full copy of the (potentially much larger) shared
+ dataframe: users with the same access scope share this key too.
"""
- Cache-key material binding cached annotation data to its security
- context.
+ if not query_obj or not query_obj.annotation_layers:
+ return None
+ return self.query_cache_key(
+ query_obj, annotation_context=self._annotation_cache_context(query_obj)
+ )
- Annotation payloads are fetched per requesting user and stored on the
- same cache entry as the dataframe, so the key also binds the requesting
- user and, for chart-backed layers, the RLS clauses of the referenced
- chart's datasource.
+ def _annotation_cache_context(self, query_obj: QueryObject) -> dict[str, Any]:
+ """
+ Cache-key material binding annotation data to its security *scope* so
+ users with the same access share a cache entry and users with a
+ different scope β or no access β never read each other's data.
+
+ * NATIVE layers: the ``can_read`` permission on ``Annotation``, the
+ only user-dependent dimension of these global records.
+ * Chart-backed (``line``/``table``) layers: see
+ :meth:`_annotation_source_scope`.
"""
- source_rls: dict[str, list[str] | None] = {}
+ context: dict[str, Any] = {}
+
+ if any(
+ layer.get("sourceType") == "NATIVE" for layer in query_obj.annotation_layers
+ ):
+ context["annotation_read"] = security_manager.can_access(
+ "can_read", "Annotation"
+ )
+
+ source_scope: dict[str, Any] = {}
for layer in query_obj.annotation_layers:
if (
layer.get("sourceType")
not in ANNOTATION_SOURCE_TYPES_WITH_CHART_REFERENCE
):
continue
layer_value = layer.get("value")
+ source_scope[str(layer_value)] = self._annotation_source_scope(layer)
+ if source_scope:
+ context["source_scope"] = source_scope
+
+ return context
+
+ def _annotation_source_scope(self, layer: dict[str, Any]) -> dict[str, Any]:
+ """
+ Access and data-identity cache-key material for one chart-backed
+ annotation layer.
+
+ ``access`` keeps a user denied the referenced chart's datasource from
+ reading an authorized user's cached payload. When the chart has a
+ saved query context, this runs the *same* authorization path
+ :meth:`get_viz_annotation_data` executes
+ (``QueryContext.raise_for_access``) rather than the coarser
+ :meth:`SecurityManager.can_access_datasource` β a requester whose
+ access comes from a dashboard/viewer-promiscuous-mode bypass (which
+ depends on the chart's own saved ``form_data``, e.g. its
+ ``slice_id``/``dashboardId``) would otherwise still fail that coarser,
+ context-free check and collapse onto the same denied scope as a
+ genuinely unauthorized requester, letting the latter read the
+ former's cached payload. ``data_key`` is the annotation chart's own
+ query cache key(s) β derived from the same, override-applied query
+ objects actually executed (see :meth:`_apply_annotation_overrides`),
+ so it captures the datasource version, RLS clauses, and any per-user
+ Jinja/virtual-dataset RLS material exactly as the live fetch would,
+ including material an override only introduces at a finer grain.
+ Reusing this logic avoids re-deriving it and automatically inherits
+ any future correctness fixes made there.
+ """
+ layer_value = layer.get("value")
+ datasource = None
+ try:
chart = (
ChartDAO.find_by_id(layer_value) if layer_value is not None else None
)
- annotation_datasource = chart.datasource if chart else None
- source_rls[str(layer.get("value"))] = (
- security_manager.get_rls_cache_key(annotation_datasource)
- if annotation_datasource
- else None
+ # resolved_datasource, not datasource: the latter is pinned to
+ # table-backed datasources and resolves to None for a
+ # semantic-view-backed chart, which would otherwise collapse
+ # every requester onto the same {access: None, data_key: None}
+ # scope below regardless of their actual access.
+ datasource = chart.resolved_datasource if chart else None
+ if chart is None or datasource is None:
+ return {"access": None, "data_key": None}
+
+ annotation_query_context = chart.get_query_context()
+ if annotation_query_context is not None:
+ self._apply_annotation_overrides(annotation_query_context, layer)
+ try:
+ annotation_query_context.raise_for_access()
+ access: Any = True
+ except SupersetSecurityException:
+ access = False
+ data_key: Any = [
+ annotation_query_context.query_cache_key(query_object)
Review Comment:
The annotation payload doesn't come from the host query's executed SQL. It's built from each annotation layer's own source chart (its saved query context, with overrides applied), and that's what `data_key` is derived from. So whether the host's samples handler clears metrics doesn't change what the annotation fetch runs. The host-side key material in there is just the shared base key, so I don't think switching to the prepared query buys anything.
##########
superset/common/query_context_processor.py:
##########
@@ -332,6 +344,26 @@ def get_df_payload_result(
# nonce reads the freshly-cached result instead of recomputing it.
self._mark_force_executed(query_obj, cache_key, cache.result_persisted)
+ # Annotation data is fetched per requesting user (and, for chart-backed
+ # layers, scoped by the referenced chart datasource's RLS), so it is
+ # resolved and cached under its own entry β independent of whether the
+ # (shareable) dataframe above was a hit or a miss β rather than forcing
+ # every viewer of the same chart onto their own full dataframe copy.
+ annotation_data: dict[str, Any] = {}
+ if query_obj and annotation_key and cache.status != QueryStatus.FAILED:
+ try:
+ annotation_data = self._get_annotation_data_cached(
Review Comment:
Good catch. The sink grabbed the first cursor regardless of which database it came from, so with a warm dataframe an annotation cursor on a different server would get recorded against the host DB. Fixed in 4190cb49a5: `notify_cursor` now passes the cursor's database and the task's sink ignores any cursor that isn't the task datasource's database. Added a test for it.
##########
superset/common/query_context_processor.py:
##########
@@ -470,72 +499,223 @@ def query_cache_key(self, query_obj: QueryObject, **kwargs: Any) -> str | None:
)
return cache_key
- def _annotation_cache_context(self, query_obj: QueryObject) -> dict[str, Any]:
+ def annotation_cache_key(self, query_obj: QueryObject) -> str | None:
"""
- Cache-key material binding cached annotation data to its security
- context.
-
- Annotation payloads are fetched per requesting user and stored on the
- same cache entry as the dataframe, so the key also binds the requesting
- user and, for chart-backed layers, the RLS clauses of the referenced
- chart's datasource.
+ Cache key for this query's annotation-layer payload, or ``None`` when
+ the query has no annotation layers or when some layer's access scope
+ could not be resolved (the payload is then never cached, see
+ :meth:`_annotation_source_scope`).
+
+ Annotation payloads are fetched under the requesting user's access
+ scope, which is a stricter security requirement than the dataframe
+ itself has. Keying them separately from :meth:`query_cache_key` keeps
+ that scoping from forcing every distinct viewer of an annotated chart
+ onto their own full copy of the (potentially much larger) shared
+ dataframe: users with the same access scope share this key too.
"""
- source_rls: dict[str, list[str] | None] = {}
- source_versions: dict[str, tuple[str, str]] = {}
- metadata_versions: dict[str, str] = {}
+ if not query_obj or not query_obj.annotation_layers:
+ return None
+ if (annotation_context := self._annotation_cache_context(query_obj)) is None:
+ return None
+ return self.query_cache_key(query_obj, annotation_context=annotation_context)
+
+ def _annotation_cache_context(
+ self, query_obj: QueryObject
+ ) -> dict[str, Any] | None:
+ """
+ Cache-key material binding annotation data to its security *scope* so
+ users with the same access share a cache entry and users with a
+ different scope β or no access β never read each other's data.
+
+ * NATIVE layers: the ``can_read`` permission on ``Annotation``, the
+ only user-dependent dimension of these global records.
+ * Chart-backed (``line``/``table``) layers: see
+ :meth:`_annotation_source_scope`, recorded per *layer* (not per
+ referenced chart) since two layers can reference the same chart
+ with different overrides, and so different scopes.
+
+ Returns ``None`` when any layer's scope cannot be resolved, meaning
+ the payload must not be cached at all.
+ """
+ context: dict[str, Any] = {}
+
+ if any(
+ layer.get("sourceType") == "NATIVE" for layer in query_obj.annotation_layers
+ ):
+ context["annotation_read"] = security_manager.can_access(
+ "can_read", "Annotation"
+ )
+
+ source_scope: list[dict[str, Any]] = []
for layer in query_obj.annotation_layers:
if (
layer.get("sourceType")
not in ANNOTATION_SOURCE_TYPES_WITH_CHART_REFERENCE
):
continue
- layer_value = layer.get("value")
+ if (scope := self._annotation_source_scope(layer)) is None:
+ return None
+ source_scope.append({"value": str(layer.get("value")), **scope})
+ if source_scope:
+ context["source_scope"] = source_scope
+
+ return context
+
+ def _annotation_source_scope(self, layer: dict[str, Any]) -> dict[str, Any] | None:
+ """
+ Access and data-identity cache-key material for one chart-backed
+ annotation layer, or ``None`` when it cannot be resolved.
+
+ ``None`` means "do not cache": a missing chart, a chart with neither
+ a saved query context nor a resolvable datasource, or any error while
+ deriving the scope would otherwise collapse an authorized and a
+ denied requester onto one shared identity, letting the latter read
+ the former's warmed payload. The caller fetches live instead.
+
+ ``access`` keeps a user denied the referenced chart's datasource from
+ reading an authorized user's cached payload. When the chart has a
+ saved query context, this runs the *same* authorization path
+ :meth:`get_viz_annotation_data` executes
+ (``QueryContext.raise_for_access``) rather than the coarser
+ :meth:`SecurityManager.can_access_datasource` β a requester whose
+ access comes from a dashboard/viewer-promiscuous-mode bypass (which
+ depends on the chart's own saved ``form_data``, e.g. its
+ ``slice_id``/``dashboardId``) would otherwise still fail that coarser,
+ context-free check and collapse onto the same denied scope as a
+ genuinely unauthorized requester, letting the latter read the
+ former's cached payload. ``data_key`` is the annotation chart's own
+ query cache key(s) β derived from the same, override-applied query
+ objects actually executed (see :meth:`_apply_annotation_overrides`),
+ so it captures the datasource version, RLS clauses, and any per-user
+ Jinja/virtual-dataset RLS material exactly as the live fetch would,
+ including material an override only introduces at a finer grain.
+ Reusing this logic avoids re-deriving it and automatically inherits
+ any future correctness fixes made there.
+ """
+ layer_value = layer.get("value")
+ try:
chart = (
ChartDAO.find_by_id(layer_value) if layer_value is not None else None
)
- annotation_datasource: Datasource | None = (
- chart.resolved_datasource if chart else None
- )
- if chart and isinstance(chart.query_context, str):
+ if chart is None:
+ return None
+
+ # The saved query context is what the live fetch executes and
+ # authorizes against, so it is consulted first, independent of
+ # the chart's own datasource pointer: a chart can have its
+ # ``datasource_id`` cleared while its saved query context still
+ # targets a valid dataset, and the live fetch would still work.
+ annotation_query_context = chart.get_query_context()
+ if annotation_query_context is not None:
+ self._apply_annotation_overrides(annotation_query_context, layer)
try:
- source: dict[str, Any] = json.loads(chart.query_context)[
- "datasource"
- ]
- annotation_datasource = DatasourceDAO.get_datasource(
- source["type"], source["id"]
- )
- except (
- ValueError,
- TypeError,
- KeyError,
- DatasourceNotFound,
- DatasourceTypeNotSupportedError,
- DatasourceValueIsIncorrect,
- ):
- annotation_datasource = None
+ annotation_query_context.raise_for_access()
+ access: Any = True
+ except SupersetSecurityException:
+ access = False
+ data_key: Any = [
+ annotation_query_context.query_cache_key(query_object)
+ for query_object in annotation_query_context.queries
+ ]
+ annotation_datasource: Any = annotation_query_context.datasource
+ else:
+ # Fall back to the RLS-clause identity when the chart has no
+ # saved query context to key on. resolved_datasource, not
+ # datasource: the latter is pinned to table-backed
+ # datasources and resolves to None for a semantic-view-backed
+ # chart.
+ datasource = chart.resolved_datasource
+ if datasource is None:
+ return None
+ access = security_manager.can_access_datasource(datasource)
+ data_key = security_manager.get_rls_cache_key(datasource)
+ annotation_datasource = datasource
+ scope: dict[str, Any] = {"access": access, "data_key": data_key}
+ # Semantic-view results are versioned by the layer's metadata
+ # generation and result discriminator, which the datasource-level
+ # key material above does not carry.
if isinstance(annotation_datasource, SemanticView):
- metadata_versions[str(layer_value)] = (
- annotation_datasource.metadata_generation
- )
- source_rls[str(layer.get("value"))] = (
- security_manager.get_rls_cache_key(annotation_datasource)
- if annotation_datasource
- else None
+ scope["metadata_version"] = annotation_datasource.metadata_generation
+ if (
+ discriminator := annotation_datasource.result_cache_discriminator
+ ) is not None:
+ scope["result_version"] = discriminator
+ except Exception: # noqa: BLE001 pylint: disable=broad-except
+ # Derivation can fail well beyond SupersetException: the DAO
+ # lookup and lazy ``datasource`` load are themselves live DB
+ # queries, and the RLS lookup / a virtual dataset's
+ # get_extra_cache_keys() renders Jinja, either of which can raise
+ # a driver/template error. None of that should ever 500 the whole
+ # chart-data request; fail closed instead. A fallback scope
+ # shared by every requester whose derivation failed would still
+ # be a shared identity, so the answer is "no caching", not a
+ # substitute key.
+ logger.warning(
+ "Could not derive annotation cache key for chart %s; "
+ "annotation data for this request will not be cached",
+ layer_value,
+ exc_info=True,
)
- if isinstance(annotation_datasource, SemanticView):
- discriminator: tuple[str, str] | None = (
- annotation_datasource.result_cache_discriminator
+ return None
+ return scope
+
+ def _get_annotation_data_cached(
+ self,
+ query_obj: QueryObject,
+ cache_key: str | None,
+ force_query: bool,
+ force_cached: bool | None,
+ timeout: int | None,
+ datasource_uid: str | None,
+ ) -> dict[str, Any]:
+ """
+ Fetch this query's annotation-layer payload, cached under its own
+ (user/RLS-scoped) entry, separate from the shared dataframe cache.
+
+ A ``None`` ``cache_key`` means some layer's access scope could not be
+ resolved (see :meth:`_annotation_source_scope`), so there is no key
+ under which a cached payload could be safely shared: an unresolved
+ scope looks the same for an authorized and a denied requester. The
+ payload is then fetched live and never cached. That holds under
+ ``force_cached`` too: computing here beats failing the read-back of
+ an otherwise successful async task.
+ """
+ if cache_key is None:
+ return self.get_annotation_data(query_obj)
Review Comment:
Agreed the uncaught error was wrong. I didn't go with returning a cache miss, because the read-back after a successful task re-probes with `force_cached` too, so an unresolved scope would miss forever and keep resubmitting tasks. Instead the async probe in `_run_async` now catches the failed/validation errors and returns the same 400 the sync path does (4190cb49a5), so B gets the 'chart not found' 400 rather than a 500.
##########
superset/common/query_context_processor.py:
##########
@@ -474,73 +503,226 @@ def query_cache_key(self, query_obj: QueryObject, **kwargs: Any) -> str | None:
)
return cache_key
- def _annotation_cache_context(self, query_obj: QueryObject) -> dict[str, Any]:
+ def annotation_cache_key(self, query_obj: QueryObject) -> str | None:
"""
- Cache-key material binding cached annotation data to its security
- context.
-
- Annotation payloads are fetched per requesting user and stored on the
- same cache entry as the dataframe, so the key also binds the requesting
- user and, for chart-backed layers, the RLS clauses of the referenced
- chart's datasource.
+ Cache key for this query's annotation-layer payload, or ``None`` when
+ the query has no annotation layers or when some layer's access scope
+ could not be resolved (the payload is then never cached, see
+ :meth:`_annotation_source_scope`).
+
+ Annotation payloads are fetched under the requesting user's access
+ scope, which is a stricter security requirement than the dataframe
+ itself has. Keying them separately from :meth:`query_cache_key` keeps
+ that scoping from forcing every distinct viewer of an annotated chart
+ onto their own full copy of the (potentially much larger) shared
+ dataframe: users with the same access scope share this key too.
"""
- source_rls: dict[str, list[str] | None] = {}
- source_versions: dict[str, tuple[str, str]] = {}
- metadata_versions: dict[str, str] = {}
+ if not query_obj or not query_obj.annotation_layers:
+ return None
+ if (annotation_context := self._annotation_cache_context(query_obj)) is None:
+ return None
+ return self.query_cache_key(query_obj, annotation_context=annotation_context)
+
+ def _annotation_cache_context(
+ self, query_obj: QueryObject
+ ) -> dict[str, Any] | None:
+ """
+ Cache-key material binding annotation data to its security *scope* so
+ users with the same access share a cache entry and users with a
+ different scope β or no access β never read each other's data.
+
+ * NATIVE layers: the ``can_read`` permission on ``Annotation``, the
+ only user-dependent dimension of these global records.
+ * Chart-backed (``line``/``table``) layers: see
+ :meth:`_annotation_source_scope`, recorded per *layer* (not per
+ referenced chart) since two layers can reference the same chart
+ with different overrides, and so different scopes.
+
+ Returns ``None`` when any layer's scope cannot be resolved, meaning
+ the payload must not be cached at all.
+ """
+ context: dict[str, Any] = {}
+
+ if any(
+ layer.get("sourceType") == "NATIVE" for layer in query_obj.annotation_layers
+ ):
+ context["annotation_read"] = security_manager.can_access(
+ "can_read", "Annotation"
+ )
+
+ source_scope: list[dict[str, Any]] = []
for layer in query_obj.annotation_layers:
if (
layer.get("sourceType")
not in ANNOTATION_SOURCE_TYPES_WITH_CHART_REFERENCE
):
continue
- layer_value = layer.get("value")
+ if (scope := self._annotation_source_scope(layer)) is None:
+ return None
+ source_scope.append({"value": str(layer.get("value")), **scope})
+ if source_scope:
+ context["source_scope"] = source_scope
+
+ return context
+
+ def _annotation_source_scope(self, layer: dict[str, Any]) -> dict[str, Any] | None:
+ """
+ Access and data-identity cache-key material for one chart-backed
+ annotation layer, or ``None`` when it cannot be resolved.
+
+ ``None`` means "do not cache": a missing chart, a chart with neither
+ a saved query context nor a resolvable datasource, or any error while
+ deriving the scope would otherwise collapse an authorized and a
+ denied requester onto one shared identity, letting the latter read
+ the former's warmed payload. The caller fetches live instead.
+
+ ``access`` keeps a user denied the referenced chart's datasource from
+ reading an authorized user's cached payload. When the chart has a
+ saved query context, this runs the *same* authorization path
+ :meth:`get_viz_annotation_data` executes
+ (``QueryContext.raise_for_access``) rather than the coarser
+ :meth:`SecurityManager.can_access_datasource` β a requester whose
+ access comes from a dashboard/viewer-promiscuous-mode bypass (which
+ depends on the chart's own saved ``form_data``, e.g. its
+ ``slice_id``/``dashboardId``) would otherwise still fail that coarser,
+ context-free check and collapse onto the same denied scope as a
+ genuinely unauthorized requester, letting the latter read the
+ former's cached payload. ``data_key`` is the annotation chart's own
+ query cache key(s) β derived from the same, override-applied query
+ objects actually executed (see :meth:`_apply_annotation_overrides`),
+ so it captures the datasource version, RLS clauses, and any per-user
+ Jinja/virtual-dataset RLS material exactly as the live fetch would,
+ including material an override only introduces at a finer grain.
+ Reusing this logic avoids re-deriving it and automatically inherits
+ any future correctness fixes made there.
+ """
+ layer_value = layer.get("value")
+ try:
chart = (
ChartDAO.find_by_id(layer_value) if layer_value is not None else None
)
- annotation_datasource: Datasource | None = (
- chart.resolved_datasource if chart else None
- )
- if chart and isinstance(chart.query_context, str):
+ if chart is None:
+ return None
+
+ # The saved query context is what the live fetch executes and
+ # authorizes against, so it is consulted first, independent of
+ # the chart's own datasource pointer: a chart can have its
+ # ``datasource_id`` cleared while its saved query context still
+ # targets a valid dataset, and the live fetch would still work.
+ annotation_query_context = chart.get_query_context()
+ if annotation_query_context is not None:
+ self._apply_annotation_overrides(annotation_query_context, layer)
try:
- source: dict[str, Any] = json.loads(chart.query_context)[
- "datasource"
- ]
- annotation_datasource = DatasourceDAO.get_datasource(
- source["type"], source["id"]
- )
- except (
- ValueError,
- TypeError,
- KeyError,
- DatasourceNotFound,
- DatasourceTypeNotSupportedError,
- DatasourceValueIsIncorrect,
- ):
- annotation_datasource = None
+ annotation_query_context.raise_for_access()
+ access: Any = True
+ except SupersetSecurityException:
+ access = False
+ data_key: Any = [
+ annotation_query_context.query_cache_key(query_object)
Review Comment:
That's real, but I think it's the same limitation the regular chart cache has. `get_rls_cache_key` only covers RLS on the datasource itself, so an adhoc subquery over another RLS-protected table isn't in any chart's `query_cache_key`, including the source chart's own. `data_key` here is exactly that key, so annotations aren't any weaker than the chart they reference. Fixing it properly means teaching the cache key about tables inside adhoc expressions, which I'd want to do as its own change rather than in this one.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] fix(charts): decouple annotation cache from the shared dataframe cache [superset]
Posted by "bito-code-review[bot] (via GitHub)" <gi...@apache.org>.
bito-code-review[bot] commented on PR #44406:
URL: https://github.com/apache/superset/pull/44406#issuecomment-5961616821
AI Code Review is in progress (usually takes 3 to 15 minutes unless it's a very large PR).
<details>
<summary>
Bito Usage Guide
</summary>
**Commands**
Type the following command in the pull request comment and save the comment.
- `/review` - Manually triggers an incremental AI Review.
- `/review full` - Manually triggers a full AI Review.
- `/pause` - Pauses automatic reviews on this pull request.
- `/resume` - Resumes automatic reviews.
- `/resolve` - Marks all Bito-posted review comments as resolved.
- `/abort` - Cancels all in-progress reviews.
Refer to the <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/available-commands" target="_blank" rel="noopener noreferrer">documentation</a> for additional commands.
**Configuration**
This repository uses `Superset` You can customize the agent settings <a href="https://alpha.bito.ai/home/ai-agents/code-review-agent" target="_blank" rel="noopener noreferrer">here</a> or contact your Bito workspace admin at evan@preset.io.
**Documentation & Help**
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/getting-started/install-run-using-bito-cloud/create-or-customize-an-agent-instance" target="_blank" rel="noopener noreferrer">Customize agent settings</a>
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/implementing-custom-code-review-rules" target="_blank" rel="noopener noreferrer">Review rules</a>
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent" target="_blank" rel="noopener noreferrer">General documentation</a>
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/faqs" target="_blank" rel="noopener noreferrer">FAQ</a>
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] fix(charts): decouple annotation cache from the shared dataframe cache [superset]
Posted by "sadpandajoe (via GitHub)" <gi...@apache.org>.
sadpandajoe commented on code in PR #44406:
URL: https://github.com/apache/superset/pull/44406#discussion_r4333697489
##########
superset/common/query_context_processor.py:
##########
@@ -454,17 +494,17 @@ def get_df_payload_result(
def query_cache_key(self, query_obj: QueryObject, **kwargs: Any) -> str | None:
"""
Returns a QueryObject cache key for objects in self.queries
+
+ This key covers the dataframe alone. It intentionally does not bind
Review Comment:
With the annotation scope removed from this key, the shared async task key (`_query_task_cache_key` in `async_queries.py`) no longer separates users whose annotation access differs, but the worker still fetches annotations as whoever submitted first. If a user who can read the host dataset but not the annotation source starts the task, an authorized user who joins it gets that task's annotation access failure on a chart they're allowed to see. Should the task identity also include the annotation key, or the annotation fetch move out of the shared task?
##########
superset/common/query_context_processor.py:
##########
@@ -478,73 +518,256 @@ def query_cache_key(self, query_obj: QueryObject, **kwargs: Any) -> str | None:
)
return cache_key
- def _annotation_cache_context(self, query_obj: QueryObject) -> dict[str, Any]:
+ def annotation_cache_key(self, query_obj: QueryObject) -> str | None:
"""
- Cache-key material binding cached annotation data to its security
- context.
-
- Annotation payloads are fetched per requesting user and stored on the
- same cache entry as the dataframe, so the key also binds the requesting
- user and, for chart-backed layers, the RLS clauses of the referenced
- chart's datasource.
+ Cache key for this query's annotation-layer payload, or ``None`` when
+ the query has no annotation layers or when some layer's access scope
+ could not be resolved (the payload is then never cached, see
+ :meth:`_annotation_source_scope`).
+
+ Annotation payloads are fetched under the requesting user's access
+ scope, which is a stricter security requirement than the dataframe
+ itself has. Keying them separately from :meth:`query_cache_key` keeps
+ that scoping from forcing every distinct viewer of an annotated chart
+ onto their own full copy of the (potentially much larger) shared
+ dataframe: users with the same access scope share this key too.
+ """
+ if not query_obj or not query_obj.annotation_layers:
+ return None
+ if (annotation_context := self._annotation_cache_context(query_obj)) is None:
+ return None
+ return self.query_cache_key(query_obj, annotation_context=annotation_context)
+
+ def _annotation_cache_context(
+ self, query_obj: QueryObject
+ ) -> dict[str, Any] | None:
+ """
+ Cache-key material binding annotation data to its security *scope* so
+ users with the same access share a cache entry and users with a
+ different scope β or no access β never read each other's data.
+
+ * NATIVE layers: the ``can_read`` permission on ``Annotation``, the
+ only user-dependent dimension of these global records.
+ * Chart-backed (``line``/``table``) layers: see
+ :meth:`_annotation_source_scope`, recorded per *layer* (not per
+ referenced chart) since two layers can reference the same chart
+ with different overrides, and so different scopes.
+
+ Returns ``None`` when any layer's scope cannot be resolved, meaning
+ the payload must not be cached at all.
"""
- source_rls: dict[str, list[str] | None] = {}
- source_versions: dict[str, tuple[str, str]] = {}
- metadata_versions: dict[str, str] = {}
+ context: dict[str, Any] = {}
+
+ if any(
+ layer.get("sourceType") == "NATIVE" for layer in query_obj.annotation_layers
+ ):
+ context["annotation_read"] = security_manager.can_access(
+ "can_read", "Annotation"
+ )
+
+ source_scope: list[dict[str, Any]] = []
for layer in query_obj.annotation_layers:
if (
layer.get("sourceType")
not in ANNOTATION_SOURCE_TYPES_WITH_CHART_REFERENCE
):
continue
- layer_value = layer.get("value")
+ if (scope := self._annotation_source_scope(layer)) is None:
+ return None
+ source_scope.append({"value": str(layer.get("value")), **scope})
+ if source_scope:
+ context["source_scope"] = source_scope
+
+ return context
+
+ def _annotation_source_scope(self, layer: dict[str, Any]) -> dict[str, Any] | None:
+ """
+ Access and data-identity cache-key material for one chart-backed
+ annotation layer, or ``None`` when it cannot be resolved.
+
+ ``None`` means "do not cache": a missing chart, a chart with neither
+ a saved query context nor a resolvable datasource, or any error while
+ deriving the scope would otherwise collapse an authorized and a
+ denied requester onto one shared identity, letting the latter read
+ the former's warmed payload. The caller fetches live instead. A
+ referenced chart whose own caching is disabled
+ (``CACHE_DISABLED_TIMEOUT``) is ``None`` too: its rows are never
+ shared through its own cache, so they must not be shared through the
+ host chart's annotation entry either.
+
+ ``access`` keeps a user denied the referenced chart's datasource from
+ reading an authorized user's cached payload. When the chart has a
+ saved query context, this runs the *same* authorization path
+ :meth:`get_viz_annotation_data` executes
+ (``QueryContext.raise_for_access``) rather than the coarser
+ :meth:`SecurityManager.can_access_datasource` β a requester whose
+ access comes from a dashboard/viewer-promiscuous-mode bypass (which
+ depends on the chart's own saved ``form_data``, e.g. its
+ ``slice_id``/``dashboardId``) would otherwise still fail that coarser,
+ context-free check and collapse onto the same denied scope as a
+ genuinely unauthorized requester, letting the latter read the
+ former's cached payload. ``data_key`` is the annotation chart's own
+ query cache key(s) β derived from the same, override-applied and
+ result-type-prepared query objects actually executed (see
+ :meth:`_apply_annotation_overrides` and
+ :func:`~superset.common.query_actions.prepare_query_object`), so it
+ captures the datasource version, RLS clauses, and any per-user
+ Jinja/virtual-dataset RLS material exactly as the live fetch would,
+ including material an override only introduces at a finer grain or
+ that a samples query's cleared metrics only expose at execution.
+ Reusing this logic avoids re-deriving it and automatically inherits
+ any future correctness fixes made there.
+ """
+ layer_value = layer.get("value")
+ try:
chart = (
ChartDAO.find_by_id(layer_value) if layer_value is not None else None
)
- annotation_datasource: Datasource | None = (
- chart.resolved_datasource if chart else None
- )
- if chart and isinstance(chart.query_context, str):
+ if chart is None:
+ return None
+
+ # The saved query context is what the live fetch executes and
+ # authorizes against, so it is consulted first, independent of
+ # the chart's own datasource pointer: a chart can have its
+ # ``datasource_id`` cleared while its saved query context still
+ # targets a valid dataset, and the live fetch would still work.
+ annotation_query_context = chart.get_query_context()
+ if annotation_query_context is not None:
+ if (
+ annotation_query_context.get_cache_timeout()
+ == CACHE_DISABLED_TIMEOUT
+ ):
+ return None
+ self._apply_annotation_overrides(annotation_query_context, layer)
try:
- source: dict[str, Any] = json.loads(chart.query_context)[
- "datasource"
+ annotation_query_context.raise_for_access()
+ access: Any = True
+ except SupersetSecurityException:
+ access = False
+ # A denied requester never reads or writes a cached payload,
+ # so skip deriving a data identity (which can render the
+ # dataset's Jinja) for a resource they cannot access.
+ # Keyed on the query object the fetch will execute: a samples
+ # or drill-detail result type rewrites the saved query (e.g.
+ # clears its metrics) before running it, and that can change
+ # the per-user Jinja material the key has to carry.
+ data_key: Any = (
+ [
+ annotation_query_context.query_cache_key(
+ prepare_query_object(annotation_query_context, query_object)
Review Comment:
`prepare_query_object(annotation_query_context, query_object)` defaults to the context's result type, but execution uses `query_object.result_type or context.result_type`. For a saved FULL context with a SAMPLES query, sampling rewrites the query (clears metrics, selects every column), so a per-user Jinja column or RLS clause that only renders in the executed query is missing from this key. The second viewer could then get the first viewer's cached annotation rows. Should this pass `query_object.result_type` as the third argument?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] fix(charts): decouple annotation cache from the shared dataframe cache [superset]
Posted by "rusackas (via GitHub)" <gi...@apache.org>.
rusackas commented on code in PR #44406:
URL: https://github.com/apache/superset/pull/44406#discussion_r4139021305
##########
tests/unit_tests/common/test_query_context_processor.py:
##########
@@ -2226,6 +2396,180 @@ def test_get_df_payload_no_warning_when_not_memory_limited() -> None:
assert result["warning"] is None
+def test_get_df_payload_result_decouples_annotation_cache_from_dataframe_cache():
Review Comment:
Good catch, added the return type hint.
##########
tests/unit_tests/common/test_query_context_processor.py:
##########
@@ -2226,6 +2396,180 @@ def test_get_df_payload_no_warning_when_not_memory_limited() -> None:
assert result["warning"] is None
+def test_get_df_payload_result_decouples_annotation_cache_from_dataframe_cache():
+ """
+ The dataframe cache entry must stay shareable across viewers, and
+ annotation-layer data must be resolved through its own (per-user) cache
+ path -- not stored on the dataframe's cache entry -- so that two viewers
+ of the same annotated chart share one dataframe cache hit while each
+ still gets their own annotation-security-scoped payload.
+ """
+ from superset.common.query_object import QueryObject
+
+ mock_query_context = MagicMock()
+ mock_query_context.force = False
+ mock_datasource = MagicMock()
+ mock_datasource.column_names = ["col1"]
+
+ processor = QueryContextProcessor(mock_query_context)
+ processor._qc_datasource = mock_datasource
+
+ query_obj = QueryObject(
+ datasource=mock_datasource,
+ columns=["col1"],
+ annotation_layers=[
+ {
+ "annotationType": "EVENT",
+ "sourceType": "NATIVE",
+ "name": "a",
+ "value": 1,
+ }
+ ],
+ )
+
+ class MockCache:
+ def __init__(self):
+ self.is_loaded = True
+ self.applied_filter_columns = ["col1"]
+ self.df = pd.DataFrame({"col1": [1, 2, 3]})
+ self.query = ""
+ self.status = "success"
+ self.cache_dttm = "2024-01-01T00:00:00"
+ self.queried_dttm = "2024-01-01T00:00:00"
+ self.stacktrace = None
+ self.error_message = None
+ self.is_cached = True
+ self.sql_rowcount = 0
+ self.cache_value = None
+ self.applied_template_filters = []
+ self.rejected_filter_columns = []
+ self.annotation_data = {"stale": "should not be used"}
+ self.bq_memory_limited = False
+ self.bq_memory_limited_row_count = 0
+ self.result_persisted = False
+ self.set_query_result = MagicMock()
+
+ mock_cache = MockCache()
+
+ with (
+ patch(
+ "superset.common.query_context_processor.QueryCacheManager"
+ ) as mock_cache_manager,
+ patch.object(query_obj, "validate", return_value=None),
+ patch.object(processor, "query_cache_key", return_value="df-key"),
+ patch.object(processor, "annotation_cache_key", return_value="ann-key"),
+ patch.object(
+ processor, "_get_annotation_data_cached", return_value={"a": [1, 2]}
+ ) as mock_get_annotation,
+ patch.object(processor, "get_cache_timeout", return_value=3600),
+ ):
+ mock_cache_manager.get.return_value = mock_cache
+ result = processor.get_df_payload(query_obj, force_cached=False)
+
+ # The dataframe cache is a hit, so the (expensive) query is never re-run,
+ # and its cache entry is never rewritten.
+ mock_cache.set_query_result.assert_not_called()
+
+ # Annotation data is resolved through the separate, per-user cache path,
+ # keyed by the dedicated annotation cache key -- not the dataframe's key.
+ mock_get_annotation.assert_called_once()
+ _, kwargs = mock_get_annotation.call_args
+ assert kwargs["cache_key"] == "ann-key"
+
+ # The payload serves the freshly-resolved annotation data, not whatever
+ # (stale) value happened to sit on the dataframe's cache object.
+ assert result["annotation_data"] == {"a": [1, 2]}
+
+
+def test_get_df_payload_result_annotation_refresh_independent_of_df_marker():
Review Comment:
Good catch, added type hints on the mock and the nested helper.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] fix(charts): decouple annotation cache from the shared dataframe cache [superset]
Posted by "sadpandajoe (via GitHub)" <gi...@apache.org>.
sadpandajoe commented on code in PR #44406:
URL: https://github.com/apache/superset/pull/44406#discussion_r4246802793
##########
superset/common/query_context_processor.py:
##########
@@ -937,6 +1119,51 @@ def get_native_annotation_data(query_obj: QueryObject) -> dict[str, Any]:
annotation_data[layer_name] = result
return annotation_data
+ @staticmethod
+ def _apply_annotation_overrides(
+ query_context: QueryContext, annotation_layer: dict[str, Any]
+ ) -> None:
+ """
+ Apply a chart-backed annotation layer's per-request overrides (time
+ grain, time range) to its saved query context's queries, in place.
+
+ Shared by :meth:`get_viz_annotation_data` (which executes the
+ overridden query) and :meth:`_annotation_source_scope` (which derives
+ its cache key from it), so the key always reflects the query that
+ actually runs: an override can introduce per-user Jinja/RLS material
+ the saved, un-overridden query lacks (e.g. a template that only calls
+ ``current_user_id()`` at a finer time grain), which deriving the key
+ from the un-overridden query would silently miss.
+ """
+ # Only the source chart's rows are used. Dropping its own annotation
+ # layers also stops charts that annotate each other (A -> B -> A)
+ # from recursing.
+ for query_object in query_context.queries:
+ query_object.annotation_layers = []
+
+ if not (overrides := annotation_layer.get("overrides")):
+ return
+
+ if time_grain_sqla := overrides.get("time_grain_sqla"):
+ for query_object in query_context.queries:
+ query_object.extras["time_grain_sqla"] = time_grain_sqla
+
+ if time_range := overrides.get("time_range"):
+ from_dttm, to_dttm = get_since_until_from_time_range(time_range)
+
+ for query_object in query_context.queries:
+ query_object.from_dttm = from_dttm
+ query_object.to_dttm = to_dttm
+ # Keep the logical range alongside the resolved bounds:
+ # ``QueryObject.cache_key`` keys on ``time_range`` when set
+ # (dropping the bounds), so a relative override such as
+ # "Last week" keeps one key as time passes instead of
+ # contributing freshly resolved timestamps on every request
+ # (which would defeat cache and force-nonce reuse), and an
+ # override on a saved query without a range of its own is
+ # still told apart from a different override.
+ query_object.time_range = time_range
Review Comment:
Assigning `query_object.time_range` here makes `QueryObject.cache_key()` drop the resolved bounds, but the bounds above are computed without the saved query's `time_shift`. For a saved annotation chart with `time_shift="1 year ago"`, a `2025-01-01 : 2025-02-01` override executes January 2025 data, while a normal run of that chart with the same range and shift executes January 2024; both now produce the same query cache key. Whichever runs first populates the entry the other reads, so the annotation can show the wrong year's rows (or the chart can show the annotation's). Should the override keep the explicit bounds in the key whenever the query carries a `time_shift`, or apply the shift to the override range the way the query factory does?
##########
tests/unit_tests/semantic_layers/simple_cache_clear_test.py:
##########
@@ -307,32 +317,24 @@ def test_annotation_result_key_uses_saved_source_generation(
from superset.common.query_context_processor import QueryContextProcessor
view: SemanticView = SemanticView(name="view", semantic_layer=layer)
- chart: Mock = Mock(query_context='{"datasource":{"id":17,"type":"semantic_view"}}')
+ chart: Mock = _saved_context_chart(view)
query: Mock = Mock(annotation_layers=[{"sourceType": "line", "value": 7}])
processor: QueryContextProcessor = QueryContextProcessor(Mock())
- get_datasource: Mock
with (
app.app_context(),
patch(
"superset.common.query_context_processor.ChartDAO.find_by_id",
return_value=chart,
),
- patch(
- "superset.daos.datasource.DatasourceDAO.get_datasource",
- return_value=view,
- ) as get_datasource,
- patch(
- "superset.common.query_context_processor.security_manager.get_rls_cache_key",
- return_value=[],
- ),
):
- first: dict[str, Any] = processor._annotation_cache_context(query)
+ first: dict[str, Any] | None = processor._annotation_cache_context(query)
+ assert first is not None
layer.cache_version = 1
session.info.clear()
view.forget_metadata()
assert processor._annotation_cache_context(query) != first
Review Comment:
This inequality also passes if scope derivation starts failing after the version rotates: `_annotation_cache_context` returns `None` on any derivation error, and `None != first` is true. A regression that silently disables annotation caching for semantic views after a metadata change would therefore still satisfy this test (the `call_count == 2` check below doesn't distinguish it either), and the same applies to the version-rotation assertion in `test_cache_keys_compose_metadata_and_producer_versions`. Could the second context be captured and asserted non-`None`, with the rotated version field compared and the unchanged one asserted equal?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] fix(charts): decouple annotation cache from the shared dataframe cache [superset]
Posted by "rusackas (via GitHub)" <gi...@apache.org>.
rusackas commented on code in PR #44406:
URL: https://github.com/apache/superset/pull/44406#discussion_r4312494299
##########
superset/common/query_context_processor.py:
##########
@@ -356,6 +361,39 @@ def get_df_payload_result(
# nonce reads the freshly-cached result instead of recomputing it.
self._mark_force_executed(query_obj, cache_key, cache.result_persisted)
+ # Annotation data is fetched per requesting user (and, for chart-backed
+ # layers, scoped by the referenced chart datasource's RLS), so it is
+ # resolved and cached under its own entry β independent of whether the
+ # (shareable) dataframe above was a hit or a miss β rather than forcing
+ # every viewer of the same chart onto their own full dataframe copy.
+ annotation_data: dict[str, Any] = {}
+ if (
+ query_obj
+ and query_obj.annotation_layers
+ and cache.status != QueryStatus.FAILED
+ ):
+ annotation_start_ns = time.perf_counter_ns()
+ try:
+ annotation_data = self._get_annotation_data_cached(
+ query_obj=query_obj,
+ cache_key=annotation_key,
+ force_query=annotation_force_query,
+ force_cached=force_cached,
+ timeout=self.get_cache_timeout(),
+ datasource_uid=self._qc_datasource.uid,
+ )
+ except SemanticResultCompletenessError:
+ raise
+ except QueryObjectValidationError as ex:
+ cache.error_message = str(ex)
Review Comment:
Good catch, switched to `QueryObjectValidationMessage(ex)` so the 400 validation error type is preserved.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] fix(charts): decouple annotation cache from the shared dataframe cache [superset]
Posted by "rusackas (via GitHub)" <gi...@apache.org>.
rusackas commented on code in PR #44406:
URL: https://github.com/apache/superset/pull/44406#discussion_r4077925923
##########
superset/common/query_context_processor.py:
##########
@@ -445,31 +464,150 @@ def query_cache_key(self, query_obj: QueryObject, **kwargs: Any) -> str | None:
)
return cache_key
- def _annotation_cache_context(self, query_obj: QueryObject) -> dict[str, Any]:
+ def annotation_cache_key(self, query_obj: QueryObject) -> str | None:
+ """
+ Cache key for this query's annotation-layer payload, or ``None`` when
+ the query has no annotation layers.
+
+ Annotation payloads are fetched under the requesting user's access
+ scope, which is a stricter security requirement than the dataframe
+ itself has. Keying them separately from :meth:`query_cache_key` keeps
+ that scoping from forcing every distinct viewer of an annotated chart
+ onto their own full copy of the (potentially much larger) shared
+ dataframe: users with the same access scope share this key too.
"""
- Cache-key material binding cached annotation data to its security
- context.
+ if not query_obj or not query_obj.annotation_layers:
+ return None
+ return self.query_cache_key(
+ query_obj, annotation_context=self._annotation_cache_context(query_obj)
+ )
- Annotation payloads are fetched per requesting user and stored on the
- same cache entry as the dataframe, so the key also binds the requesting
- user and, for chart-backed layers, the RLS clauses of the referenced
- chart's datasource.
+ def _annotation_cache_context(self, query_obj: QueryObject) -> dict[str, Any]:
+ """
+ Cache-key material binding annotation data to its security *scope* so
+ users with the same access share a cache entry and users with a
+ different scope β or no access β never read each other's data.
+
+ * NATIVE layers: the ``can_read`` permission on ``Annotation``, the
+ only user-dependent dimension of these global records.
+ * Chart-backed (``line``/``table``) layers: see
+ :meth:`_annotation_source_scope`.
"""
- source_rls: dict[str, list[str] | None] = {}
+ context: dict[str, Any] = {}
+
+ if any(
+ layer.get("sourceType") == "NATIVE" for layer in query_obj.annotation_layers
+ ):
+ context["annotation_read"] = security_manager.can_access(
+ "can_read", "Annotation"
+ )
+
+ source_scope: dict[str, Any] = {}
for layer in query_obj.annotation_layers:
if layer.get("sourceType") not in ("line", "table"):
continue
layer_value = layer.get("value")
- chart = (
- ChartDAO.find_by_id(layer_value) if layer_value is not None else None
+ source_scope[str(layer_value)] = self._annotation_source_scope(layer_value)
+ if source_scope:
+ context["source_scope"] = source_scope
+
+ return context
+
+ def _annotation_source_scope(self, layer_value: Any) -> dict[str, Any]:
+ """
+ Access and data-identity cache-key material for one chart-backed
+ annotation layer.
+
+ ``access`` keeps a user denied the referenced chart's datasource from
+ reading an authorized user's cached payload. ``data_key`` is the
+ annotation chart's own query cache key(s), which already capture the
+ datasource version, RLS clauses, and any per-user Jinja/virtual-dataset
+ RLS material β reusing it here avoids re-deriving that logic and
+ automatically inherits any future correctness fixes made there.
+ """
+ chart = ChartDAO.find_by_id(layer_value) if layer_value is not None else None
+ datasource = chart.datasource if chart else None
Review Comment:
Good catch. Swapped `chart.datasource` for `chart.resolved_datasource` here so semantic-view-backed annotation layers get their own access-scoped cache key instead of sharing one across every requester regardless of access.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] fix(charts): decouple annotation cache from the shared dataframe cache [superset]
Posted by "bito-code-review[bot] (via GitHub)" <gi...@apache.org>.
bito-code-review[bot] commented on code in PR #44406:
URL: https://github.com/apache/superset/pull/44406#discussion_r4172344880
##########
tests/unit_tests/common/test_query_context_processor.py:
##########
@@ -117,25 +118,259 @@ def processor(mock_query_context):
return processor
-def test_query_cache_key_binds_annotation_data_to_requesting_user(processor):
- """The cache key for annotated queries must differ per requesting user."""
+def test_annotation_cache_key_binds_native_annotation_read_scope(processor) -> None:
+ """The annotation cache key for NATIVE layers must differ when the
+ requester's ``can_read`` (Annotation) access differs -- not who they are."""
query_obj = MagicMock()
query_obj.annotation_layers = [{"sourceType": "NATIVE", "name": "a", "value": 1}]
- with (
- patch(
- "superset.common.query_context_processor.get_user_id",
- side_effect=[1, 2],
- ),
- patch("superset.common.query_context_processor.security_manager"),
- ):
- processor.query_cache_key(query_obj)
- processor.query_cache_key(query_obj)
+ # ``security_manager`` autodetects as an async spec under a bare
+ # ``patch()`` (its real object trips ``unittest.mock``'s coroutine
+ # inference), which would silently turn every attribute access into an
+ # ``AsyncMock`` returning a fresh unawaited coroutine per call -- always
+ # unequal to itself and never equal to a configured return value. Forcing
+ # ``new_callable=MagicMock`` keeps these synchronous, as the real object
+ # is.
+ with patch(
+ "superset.common.query_context_processor.security_manager",
+ new_callable=MagicMock,
+ ) as security_manager:
+ security_manager.can_access.side_effect = [True, False]
+ processor.annotation_cache_key(query_obj)
+ processor.annotation_cache_key(query_obj)
contexts = [
call.kwargs["annotation_context"] for call in query_obj.cache_key.call_args_list
]
assert contexts[0] != contexts[1]
+def test_annotation_cache_key_shares_across_same_access_scope() -> None:
+ """Two distinct requesters (separate processor/query-object instances,
+ standing in for two different requests) with identical access scope must
+ produce identical annotation-context material. Reusing a single
+ processor/query_obj across both calls (as this test previously did)
+ would pass trivially regardless of whether the key is scope-based or
+ identity-based, since nothing about "who's asking" would ever vary."""
+ layer = {"sourceType": "NATIVE", "name": "a", "value": 1}
+ processor_a = QueryContextProcessor(MagicMock())
+ processor_b = QueryContextProcessor(MagicMock())
+ query_obj_a = MagicMock(annotation_layers=[layer])
+ query_obj_b = MagicMock(annotation_layers=[layer])
+
+ with patch(
+ "superset.common.query_context_processor.security_manager",
+ new_callable=MagicMock,
+ ) as security_manager:
+ security_manager.can_access.return_value = True
+ context_a = processor_a._annotation_cache_context(query_obj_a)
+ context_b = processor_b._annotation_cache_context(query_obj_b)
+
+ assert context_a == context_b
+
+
+def test_query_cache_key_does_not_bind_annotation_scope(processor) -> None:
+ """The dataframe cache key must stay shared across viewers of the same
+ chart, even when the query has annotation layers β only the separate
+ annotation cache key (see above) carries access-scope material."""
+ query_obj = MagicMock()
+ query_obj.annotation_layers = [{"sourceType": "NATIVE", "name": "a", "value": 1}]
+ with patch(
+ "superset.common.query_context_processor.security_manager",
+ new_callable=MagicMock,
+ ):
+ processor.query_cache_key(query_obj)
+ processor.query_cache_key(query_obj)
+ for call in query_obj.cache_key.call_args_list:
+ assert "annotation_context" not in call.kwargs
+
+
+@pytest.fixture
+def mock_annotation_chart() -> Iterator[MagicMock]:
+ """A found chart, wired as the referenced chart for
+ ``_annotation_source_scope`` tests -- factors out the repeated
+ ``ChartDAO.find_by_id`` patch those tests all need."""
+ chart = MagicMock()
+ with patch(
+ "superset.common.query_context_processor.ChartDAO.find_by_id",
+ return_value=chart,
+ ):
+ yield chart
+
+
+def test_annotation_source_scope_binds_datasource_access(
+ processor, mock_annotation_chart
+) -> None:
+ """A chart-backed annotation layer's scope must differ when the
+ requester's access to the referenced datasource differs."""
+ mock_annotation_chart.get_query_context.return_value = None
+ with patch(
+ "superset.common.query_context_processor.security_manager",
+ new_callable=MagicMock,
+ ) as security_manager:
+ security_manager.can_access_datasource.side_effect = [True, False]
+ security_manager.get_rls_cache_key.return_value = []
+ scope_a = processor._annotation_source_scope({"value": 1})
+ scope_b = processor._annotation_source_scope({"value": 1})
+ assert scope_a != scope_b
+ assert scope_a["access"] is True
+ assert scope_b["access"] is False
+
+
+def test_annotation_source_scope_reuses_referenced_chart_cache_key(
+ processor, mock_annotation_chart
+) -> None:
+ """When the referenced chart has a saved query context, its own cache
+ key(s) -- covering RLS and per-user Jinja/virtual-dataset material -- are
+ reused rather than re-derived."""
+ mock_query_object = MagicMock()
+ mock_query_context = MagicMock()
Review Comment:
<!-- Bito Reply -->
The suggestion to add `: MagicMock` type annotations to the mock variables has been addressed in the latest push. This improves type visibility for static checkers and readers, as requested.
##########
tests/unit_tests/common/test_query_context_processor.py:
##########
@@ -117,25 +118,259 @@ def processor(mock_query_context):
return processor
-def test_query_cache_key_binds_annotation_data_to_requesting_user(processor):
- """The cache key for annotated queries must differ per requesting user."""
+def test_annotation_cache_key_binds_native_annotation_read_scope(processor) -> None:
+ """The annotation cache key for NATIVE layers must differ when the
+ requester's ``can_read`` (Annotation) access differs -- not who they are."""
query_obj = MagicMock()
query_obj.annotation_layers = [{"sourceType": "NATIVE", "name": "a", "value": 1}]
- with (
- patch(
- "superset.common.query_context_processor.get_user_id",
- side_effect=[1, 2],
- ),
- patch("superset.common.query_context_processor.security_manager"),
- ):
- processor.query_cache_key(query_obj)
- processor.query_cache_key(query_obj)
+ # ``security_manager`` autodetects as an async spec under a bare
+ # ``patch()`` (its real object trips ``unittest.mock``'s coroutine
+ # inference), which would silently turn every attribute access into an
+ # ``AsyncMock`` returning a fresh unawaited coroutine per call -- always
+ # unequal to itself and never equal to a configured return value. Forcing
+ # ``new_callable=MagicMock`` keeps these synchronous, as the real object
+ # is.
+ with patch(
+ "superset.common.query_context_processor.security_manager",
+ new_callable=MagicMock,
+ ) as security_manager:
+ security_manager.can_access.side_effect = [True, False]
+ processor.annotation_cache_key(query_obj)
+ processor.annotation_cache_key(query_obj)
contexts = [
call.kwargs["annotation_context"] for call in query_obj.cache_key.call_args_list
]
assert contexts[0] != contexts[1]
+def test_annotation_cache_key_shares_across_same_access_scope() -> None:
+ """Two distinct requesters (separate processor/query-object instances,
+ standing in for two different requests) with identical access scope must
+ produce identical annotation-context material. Reusing a single
+ processor/query_obj across both calls (as this test previously did)
+ would pass trivially regardless of whether the key is scope-based or
+ identity-based, since nothing about "who's asking" would ever vary."""
+ layer = {"sourceType": "NATIVE", "name": "a", "value": 1}
+ processor_a = QueryContextProcessor(MagicMock())
+ processor_b = QueryContextProcessor(MagicMock())
+ query_obj_a = MagicMock(annotation_layers=[layer])
+ query_obj_b = MagicMock(annotation_layers=[layer])
+
+ with patch(
+ "superset.common.query_context_processor.security_manager",
+ new_callable=MagicMock,
+ ) as security_manager:
+ security_manager.can_access.return_value = True
+ context_a = processor_a._annotation_cache_context(query_obj_a)
+ context_b = processor_b._annotation_cache_context(query_obj_b)
+
+ assert context_a == context_b
+
+
+def test_query_cache_key_does_not_bind_annotation_scope(processor) -> None:
+ """The dataframe cache key must stay shared across viewers of the same
+ chart, even when the query has annotation layers β only the separate
+ annotation cache key (see above) carries access-scope material."""
+ query_obj = MagicMock()
+ query_obj.annotation_layers = [{"sourceType": "NATIVE", "name": "a", "value": 1}]
+ with patch(
+ "superset.common.query_context_processor.security_manager",
+ new_callable=MagicMock,
+ ):
+ processor.query_cache_key(query_obj)
+ processor.query_cache_key(query_obj)
+ for call in query_obj.cache_key.call_args_list:
+ assert "annotation_context" not in call.kwargs
+
+
+@pytest.fixture
+def mock_annotation_chart() -> Iterator[MagicMock]:
+ """A found chart, wired as the referenced chart for
+ ``_annotation_source_scope`` tests -- factors out the repeated
+ ``ChartDAO.find_by_id`` patch those tests all need."""
+ chart = MagicMock()
+ with patch(
+ "superset.common.query_context_processor.ChartDAO.find_by_id",
+ return_value=chart,
+ ):
+ yield chart
+
+
+def test_annotation_source_scope_binds_datasource_access(
+ processor, mock_annotation_chart
+) -> None:
+ """A chart-backed annotation layer's scope must differ when the
+ requester's access to the referenced datasource differs."""
+ mock_annotation_chart.get_query_context.return_value = None
+ with patch(
+ "superset.common.query_context_processor.security_manager",
+ new_callable=MagicMock,
+ ) as security_manager:
+ security_manager.can_access_datasource.side_effect = [True, False]
+ security_manager.get_rls_cache_key.return_value = []
+ scope_a = processor._annotation_source_scope({"value": 1})
+ scope_b = processor._annotation_source_scope({"value": 1})
+ assert scope_a != scope_b
+ assert scope_a["access"] is True
+ assert scope_b["access"] is False
+
+
+def test_annotation_source_scope_reuses_referenced_chart_cache_key(
+ processor, mock_annotation_chart
+) -> None:
+ """When the referenced chart has a saved query context, its own cache
+ key(s) -- covering RLS and per-user Jinja/virtual-dataset material -- are
+ reused rather than re-derived."""
+ mock_query_object = MagicMock()
+ mock_query_context = MagicMock()
+ mock_query_context.queries = [mock_query_object]
+ mock_query_context.query_cache_key.return_value = "referenced-chart-key"
+ mock_annotation_chart.get_query_context.return_value = mock_query_context
+ with patch(
+ "superset.common.query_context_processor.security_manager",
+ new_callable=MagicMock,
+ ) as security_manager:
+ security_manager.can_access_datasource.return_value = True
+ scope = processor._annotation_source_scope({"value": 1})
+ assert scope == {"access": True, "data_key": ["referenced-chart-key"]}
+ mock_query_context.query_cache_key.assert_called_once_with(mock_query_object)
+
+
+def test_annotation_source_scope_uses_live_fetch_authorization(
+ processor, mock_annotation_chart
+) -> None:
Review Comment:
<!-- Bito Reply -->
The user has addressed the reviewer's suggestion regarding missing type annotations for the fixture-injected parameters `processor` and `mock_annotation_chart` in the latest push. The updated code now includes the necessary type annotations, resolving the issue identified in the review.
**tests/unit_tests/common/test_query_context_processor.py**
```
def test_annotation_source_scope_binds_datasource_access(
processor: QueryContextProcessor, mock_annotation_chart: MagicMock
) -> None:
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] fix(charts): decouple annotation cache from the shared dataframe cache [superset]
Posted by "sadpandajoe (via GitHub)" <gi...@apache.org>.
sadpandajoe commented on code in PR #44406:
URL: https://github.com/apache/superset/pull/44406#discussion_r4341302255
##########
superset/common/query_context_processor.py:
##########
@@ -478,73 +518,261 @@ def query_cache_key(self, query_obj: QueryObject, **kwargs: Any) -> str | None:
)
return cache_key
- def _annotation_cache_context(self, query_obj: QueryObject) -> dict[str, Any]:
+ def annotation_cache_key(self, query_obj: QueryObject) -> str | None:
"""
- Cache-key material binding cached annotation data to its security
- context.
-
- Annotation payloads are fetched per requesting user and stored on the
- same cache entry as the dataframe, so the key also binds the requesting
- user and, for chart-backed layers, the RLS clauses of the referenced
- chart's datasource.
+ Cache key for this query's annotation-layer payload, or ``None`` when
+ the query has no annotation layers or when some layer's access scope
+ could not be resolved (the payload is then never cached, see
+ :meth:`_annotation_source_scope`).
+
+ Annotation payloads are fetched under the requesting user's access
+ scope, which is a stricter security requirement than the dataframe
+ itself has. Keying them separately from :meth:`query_cache_key` keeps
+ that scoping from forcing every distinct viewer of an annotated chart
+ onto their own full copy of the (potentially much larger) shared
+ dataframe: users with the same access scope share this key too.
+ """
+ if not query_obj or not query_obj.annotation_layers:
+ return None
+ if (annotation_context := self._annotation_cache_context(query_obj)) is None:
+ return None
+ return self.query_cache_key(query_obj, annotation_context=annotation_context)
+
+ def _annotation_cache_context(
+ self, query_obj: QueryObject
+ ) -> dict[str, Any] | None:
+ """
+ Cache-key material binding annotation data to its security *scope* so
+ users with the same access share a cache entry and users with a
+ different scope β or no access β never read each other's data.
+
+ * NATIVE layers: the ``can_read`` permission on ``Annotation``, the
+ only user-dependent dimension of these global records.
+ * Chart-backed (``line``/``table``) layers: see
+ :meth:`_annotation_source_scope`, recorded per *layer* (not per
+ referenced chart) since two layers can reference the same chart
+ with different overrides, and so different scopes.
+
+ Returns ``None`` when any layer's scope cannot be resolved, meaning
+ the payload must not be cached at all.
"""
- source_rls: dict[str, list[str] | None] = {}
- source_versions: dict[str, tuple[str, str]] = {}
- metadata_versions: dict[str, str] = {}
+ context: dict[str, Any] = {}
+
+ if any(
+ layer.get("sourceType") == "NATIVE" for layer in query_obj.annotation_layers
+ ):
+ context["annotation_read"] = security_manager.can_access(
+ "can_read", "Annotation"
+ )
+
+ source_scope: list[dict[str, Any]] = []
for layer in query_obj.annotation_layers:
if (
layer.get("sourceType")
not in ANNOTATION_SOURCE_TYPES_WITH_CHART_REFERENCE
):
continue
- layer_value = layer.get("value")
+ if (scope := self._annotation_source_scope(layer)) is None:
+ return None
+ source_scope.append({"value": str(layer.get("value")), **scope})
+ if source_scope:
+ context["source_scope"] = source_scope
+
+ return context
+
+ def _annotation_source_scope(self, layer: dict[str, Any]) -> dict[str, Any] | None:
+ """
+ Access and data-identity cache-key material for one chart-backed
+ annotation layer, or ``None`` when it cannot be resolved.
+
+ ``None`` means "do not cache": a missing chart, a chart with neither
+ a saved query context nor a resolvable datasource, or any error while
+ deriving the scope would otherwise collapse an authorized and a
+ denied requester onto one shared identity, letting the latter read
+ the former's warmed payload. The caller fetches live instead. A
+ referenced chart whose own caching is disabled
+ (``CACHE_DISABLED_TIMEOUT``) is ``None`` too: its rows are never
+ shared through its own cache, so they must not be shared through the
+ host chart's annotation entry either.
+
+ ``access`` keeps a user denied the referenced chart's datasource from
+ reading an authorized user's cached payload. When the chart has a
+ saved query context, this runs the *same* authorization path
+ :meth:`get_viz_annotation_data` executes
+ (``QueryContext.raise_for_access``) rather than the coarser
+ :meth:`SecurityManager.can_access_datasource` β a requester whose
+ access comes from a dashboard/viewer-promiscuous-mode bypass (which
+ depends on the chart's own saved ``form_data``, e.g. its
+ ``slice_id``/``dashboardId``) would otherwise still fail that coarser,
+ context-free check and collapse onto the same denied scope as a
+ genuinely unauthorized requester, letting the latter read the
+ former's cached payload. ``data_key`` is the annotation chart's own
+ query cache key(s) β derived from the same, override-applied and
+ result-type-prepared query objects actually executed (see
+ :meth:`_apply_annotation_overrides` and
+ :func:`~superset.common.query_actions.prepare_query_object`), so it
+ captures the datasource version, RLS clauses, and any per-user
+ Jinja/virtual-dataset RLS material exactly as the live fetch would,
+ including material an override only introduces at a finer grain or
+ that a samples query's cleared metrics only expose at execution.
+ Reusing this logic avoids re-deriving it and automatically inherits
+ any future correctness fixes made there.
+ """
+ layer_value = layer.get("value")
+ try:
chart = (
ChartDAO.find_by_id(layer_value) if layer_value is not None else None
)
- annotation_datasource: Datasource | None = (
- chart.resolved_datasource if chart else None
- )
- if chart and isinstance(chart.query_context, str):
+ if chart is None:
+ return None
+
+ # The saved query context is what the live fetch executes and
+ # authorizes against, so it is consulted first, independent of
+ # the chart's own datasource pointer: a chart can have its
+ # ``datasource_id`` cleared while its saved query context still
+ # targets a valid dataset, and the live fetch would still work.
+ annotation_query_context = chart.get_query_context()
+ if annotation_query_context is not None:
+ if (
+ annotation_query_context.get_cache_timeout()
+ == CACHE_DISABLED_TIMEOUT
+ ):
+ return None
+ self._apply_annotation_overrides(annotation_query_context, layer)
try:
- source: dict[str, Any] = json.loads(chart.query_context)[
- "datasource"
+ annotation_query_context.raise_for_access()
+ access: Any = True
+ except SupersetSecurityException:
+ access = False
+ # A denied requester never reads or writes a cached payload,
+ # so skip deriving a data identity (which can render the
+ # dataset's Jinja) for a resource they cannot access.
+ # Keyed on the query object the fetch will execute: a samples
+ # or drill-detail result type rewrites the saved query (e.g.
+ # clears its metrics) before running it, and that can change
+ # the per-user Jinja material the key has to carry.
+ data_key: Any = (
+ [
+ annotation_query_context.query_cache_key(
+ prepare_query_object(
+ annotation_query_context,
+ query_object,
+ query_object.result_type
+ or annotation_query_context.result_type,
+ )
+ )
+ for query_object in annotation_query_context.queries
]
- annotation_datasource = DatasourceDAO.get_datasource(
- source["type"], source["id"]
- )
- except (
- ValueError,
- TypeError,
- KeyError,
- DatasourceNotFound,
- DatasourceTypeNotSupportedError,
- DatasourceValueIsIncorrect,
+ if access
+ else None
+ )
+ annotation_datasource: Any = annotation_query_context.datasource
+ else:
+ # Fall back to the RLS-clause identity when the chart has no
+ # saved query context to key on. resolved_datasource, not
+ # datasource: the latter is pinned to table-backed
+ # datasources and resolves to None for a semantic-view-backed
+ # chart.
+ datasource = chart.resolved_datasource
+ if datasource is None:
+ return None
+ if CACHE_DISABLED_TIMEOUT in (
+ chart.cache_timeout,
+ datasource.cache_timeout,
):
- annotation_datasource = None
+ return None
+ access = security_manager.can_access_datasource(datasource)
+ data_key = security_manager.get_rls_cache_key(datasource)
+ annotation_datasource = datasource
+ scope: dict[str, Any] = {"access": access, "data_key": data_key}
+ # Semantic-view results are versioned by the layer's metadata
+ # generation and result discriminator, which the datasource-level
+ # key material above does not carry.
if isinstance(annotation_datasource, SemanticView):
- metadata_versions[str(layer_value)] = (
- annotation_datasource.metadata_generation
- )
- source_rls[str(layer.get("value"))] = (
- security_manager.get_rls_cache_key(annotation_datasource)
- if annotation_datasource
- else None
+ scope["metadata_version"] = annotation_datasource.metadata_generation
+ # Separates entries written before NULL filter selections were
+ # preserved from those written after.
+ scope["semantic_filter_protocol"] = "semantic-null-filters-v1"
+ if (
+ discriminator := annotation_datasource.result_cache_discriminator
+ ) is not None:
+ scope["result_version"] = discriminator
+ except Exception: # noqa: BLE001 pylint: disable=broad-except
+ # Derivation can fail well beyond SupersetException: the DAO
+ # lookup and lazy ``datasource`` load are themselves live DB
+ # queries, and the RLS lookup / a virtual dataset's
+ # get_extra_cache_keys() renders Jinja, either of which can raise
+ # a driver/template error. None of that should ever 500 the whole
+ # chart-data request; fail closed instead. A fallback scope
+ # shared by every requester whose derivation failed would still
+ # be a shared identity, so the answer is "no caching", not a
+ # substitute key.
+ logger.warning(
+ "Could not derive annotation cache key for chart %s; "
+ "annotation data for this request will not be cached",
+ layer_value,
+ exc_info=True,
)
- if isinstance(annotation_datasource, SemanticView):
- discriminator: tuple[str, str] | None = (
- annotation_datasource.result_cache_discriminator
+ return None
+ return scope
+
+ def _get_annotation_data_cached(
+ self,
+ query_obj: QueryObject,
+ cache_key: str | None,
+ force_query: bool,
+ force_cached: bool | None,
+ timeout: int | None,
+ datasource_uid: str | None,
+ ) -> dict[str, Any]:
+ """
+ Fetch this query's annotation-layer payload, cached under its own
+ (user/RLS-scoped) entry, separate from the shared dataframe cache.
+
+ A ``None`` ``cache_key`` means some layer's access scope could not be
+ resolved (see :meth:`_annotation_source_scope`), so there is no key
+ under which a cached payload could be safely shared: an unresolved
+ scope looks the same for an authorized and a denied requester. The
+ payload is then fetched live and never cached. That holds under
+ ``force_cached`` too: computing here beats failing the read-back of
+ an otherwise successful async task.
+ """
+ if cache_key is None:
Review Comment:
When an annotation layer's scope is `None` (including the new cache-disabled case where the referenced chart has `cache_timeout=-1`), this fetches the annotation live even under `force_cached`, and never caches it. In async mode that means a warm-dataframe load runs the source chart's query inside the HTTP request in `_run_async`'s probe, and on a cold load the worker's annotation run is discarded and the `enableAsyncMode: false` read-back runs it again in the request, with no task timeout or cancellation. A slow source chart with caching disabled can therefore hit the web request timeout even though the task succeeded. Would it make sense to return a cache miss from the async probe for uncacheable layers, or otherwise skip the in-request fetch when running async?
##########
tests/unit_tests/common/test_query_context_processor.py:
##########
@@ -126,23 +135,471 @@ def processor(mock_query_context):
return processor
-def test_query_cache_key_binds_annotation_data_to_requesting_user(processor):
- """The cache key for annotated queries must differ per requesting user."""
+def test_annotation_cache_key_binds_native_annotation_read_scope(processor) -> None:
+ """The annotation cache key for NATIVE layers must differ when the
+ requester's ``can_read`` (Annotation) access differs -- not who they are."""
query_obj = MagicMock()
query_obj.annotation_layers = [{"sourceType": "NATIVE", "name": "a", "value": 1}]
+ # ``security_manager`` autodetects as an async spec under a bare
+ # ``patch()`` (its real object trips ``unittest.mock``'s coroutine
+ # inference), which would silently turn every attribute access into an
+ # ``AsyncMock`` returning a fresh unawaited coroutine per call -- always
+ # unequal to itself and never equal to a configured return value. Forcing
+ # ``new_callable=MagicMock`` keeps these synchronous, as the real object
+ # is.
+ with patch(
+ "superset.common.query_context_processor.security_manager",
+ new_callable=MagicMock,
+ ) as security_manager:
+ security_manager.can_access.side_effect = [True, False]
+ processor.annotation_cache_key(query_obj)
+ processor.annotation_cache_key(query_obj)
+ contexts = [
+ call.kwargs["annotation_context"] for call in query_obj.cache_key.call_args_list
+ ]
+ assert contexts[0] != contexts[1]
+
+
+def test_annotation_cache_key_shares_across_same_access_scope() -> None:
+ """Two distinct requesters (separate processor/query-object instances,
+ standing in for two different requests) with identical access scope must
+ produce identical annotation-context material. Reusing a single
+ processor/query_obj across both calls (as this test previously did)
+ would pass trivially regardless of whether the key is scope-based or
+ identity-based, since nothing about "who's asking" would ever vary."""
+ layer = {"sourceType": "NATIVE", "name": "a", "value": 1}
+ processor_a = QueryContextProcessor(MagicMock())
+ processor_b = QueryContextProcessor(MagicMock())
+ query_obj_a = MagicMock(annotation_layers=[layer])
+ query_obj_b = MagicMock(annotation_layers=[layer])
+
+ # The requesting user differs between the two calls. Binding user identity
+ # into the context (by name, from either module) would make the contexts
+ # diverge, so equality below proves the key is scope-based.
with (
+ patch(
+ "superset.common.query_context_processor.security_manager",
+ new_callable=MagicMock,
+ ) as security_manager,
patch(
"superset.common.query_context_processor.get_user_id",
+ create=True,
side_effect=[1, 2],
),
- patch("superset.common.query_context_processor.security_manager"),
+ patch("superset.utils.core.get_user_id", side_effect=[1, 2]),
+ ):
+ security_manager.can_access.return_value = True
+ context_a = processor_a._annotation_cache_context(query_obj_a)
+ context_b = processor_b._annotation_cache_context(query_obj_b)
+
+ assert context_a == context_b
+
+
+def test_query_cache_key_does_not_bind_annotation_scope(processor) -> None:
+ """The dataframe cache key must stay shared across viewers of the same
+ chart, even when the query has annotation layers β only the separate
+ annotation cache key (see above) carries access-scope material."""
+ query_obj = MagicMock()
+ query_obj.annotation_layers = [{"sourceType": "NATIVE", "name": "a", "value": 1}]
+ with patch(
+ "superset.common.query_context_processor.security_manager",
+ new_callable=MagicMock,
):
processor.query_cache_key(query_obj)
processor.query_cache_key(query_obj)
- contexts = [
- call.kwargs["annotation_context"] for call in query_obj.cache_key.call_args_list
+ for call in query_obj.cache_key.call_args_list:
+ assert "annotation_context" not in call.kwargs
+
+
+@pytest.fixture
+def mock_annotation_chart() -> Iterator[MagicMock]:
+ """A found chart, wired as the referenced chart for
+ ``_annotation_source_scope`` tests -- factors out the repeated
+ ``ChartDAO.find_by_id`` patch those tests all need."""
+ chart = MagicMock()
+ with patch(
+ "superset.common.query_context_processor.ChartDAO.find_by_id",
+ return_value=chart,
+ ):
+ yield chart
+
+
+def test_annotation_source_scope_binds_datasource_access(
+ processor: QueryContextProcessor, mock_annotation_chart: MagicMock
+) -> None:
+ """A chart-backed annotation layer's scope must differ when the
+ requester's access to the referenced datasource differs."""
+ mock_annotation_chart.get_query_context.return_value = None
+ with patch(
+ "superset.common.query_context_processor.security_manager",
+ new_callable=MagicMock,
+ ) as security_manager:
+ security_manager.can_access_datasource.side_effect = [True, False]
+ security_manager.get_rls_cache_key.return_value = []
+ scope_a = processor._annotation_source_scope({"value": 1})
+ scope_b = processor._annotation_source_scope({"value": 1})
+ assert scope_a is not None
+ assert scope_b is not None
+ assert scope_a != scope_b
+ assert scope_a["access"] is True
+ assert scope_b["access"] is False
+
+
+def test_annotation_source_scope_reuses_referenced_chart_cache_key(
+ processor: QueryContextProcessor, mock_annotation_chart: MagicMock
+) -> None:
+ """When the referenced chart has a saved query context, its own cache
+ key(s) -- covering RLS and per-user Jinja/virtual-dataset material -- are
+ reused rather than re-derived."""
+ mock_query_object: MagicMock = MagicMock()
+ mock_query_context: MagicMock = MagicMock()
+ mock_query_context.queries = [mock_query_object]
+ mock_query_context.query_cache_key.return_value = "referenced-chart-key"
+ mock_annotation_chart.get_query_context.return_value = mock_query_context
+ with patch(
+ "superset.common.query_context_processor.security_manager",
+ new_callable=MagicMock,
+ ) as security_manager:
+ security_manager.can_access_datasource.return_value = True
+ scope = processor._annotation_source_scope({"value": 1})
+ assert scope == {"access": True, "data_key": ["referenced-chart-key"]}
+ mock_query_context.query_cache_key.assert_called_once_with(mock_query_object)
+
+
+def test_annotation_source_scope_keys_on_result_type_prepared_query(
+ processor: QueryContextProcessor, mock_annotation_chart: MagicMock
+) -> None:
+ """A referenced chart saved with a samples result type executes a
+ rewritten query (metrics cleared, every column selected), so its key
+ must be derived from that prepared query, not the saved one -- a Jinja
+ clause that only renders user-specific material without metrics would
+ otherwise give two viewers one key over different rows."""
+ from superset.common.query_object import QueryObject
+
+ mock_datasource = MagicMock()
+ mock_datasource.columns = []
+ saved_query = QueryObject(
+ datasource=mock_datasource, metrics=["count"], columns=["col1"]
+ )
+ mock_query_context = MagicMock()
+ mock_query_context.result_type = ChartDataResultType.SAMPLES
Review Comment:
This test sets `SAMPLES` on the context but leaves `saved_query.result_type` as `None`, so it passes even if the `query_object.result_type or annotation_query_context.result_type` argument to `prepare_query_object` is reverted to the context-only value. That override is the latest fix, and a saved `FULL` context with an individual `SAMPLES` query (metrics `["count"]`) is not exercised. Could you add a case with a `FULL` context and a `SAMPLES` query-level result type asserting `data_key` is derived from the cleared-metrics query?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] fix(charts): decouple annotation cache from the shared dataframe cache [superset]
Posted by "bito-code-review[bot] (via GitHub)" <gi...@apache.org>.
bito-code-review[bot] commented on code in PR #44406:
URL: https://github.com/apache/superset/pull/44406#discussion_r4133634486
##########
tests/unit_tests/common/test_query_context_processor.py:
##########
@@ -2226,6 +2396,180 @@ def test_get_df_payload_no_warning_when_not_memory_limited() -> None:
assert result["warning"] is None
+def test_get_df_payload_result_decouples_annotation_cache_from_dataframe_cache():
+ """
+ The dataframe cache entry must stay shareable across viewers, and
+ annotation-layer data must be resolved through its own (per-user) cache
+ path -- not stored on the dataframe's cache entry -- so that two viewers
+ of the same annotated chart share one dataframe cache hit while each
+ still gets their own annotation-security-scoped payload.
+ """
+ from superset.common.query_object import QueryObject
+
+ mock_query_context = MagicMock()
+ mock_query_context.force = False
+ mock_datasource = MagicMock()
+ mock_datasource.column_names = ["col1"]
+
+ processor = QueryContextProcessor(mock_query_context)
+ processor._qc_datasource = mock_datasource
+
+ query_obj = QueryObject(
+ datasource=mock_datasource,
+ columns=["col1"],
+ annotation_layers=[
+ {
+ "annotationType": "EVENT",
+ "sourceType": "NATIVE",
+ "name": "a",
+ "value": 1,
+ }
+ ],
+ )
+
+ class MockCache:
+ def __init__(self):
+ self.is_loaded = True
+ self.applied_filter_columns = ["col1"]
+ self.df = pd.DataFrame({"col1": [1, 2, 3]})
+ self.query = ""
+ self.status = "success"
+ self.cache_dttm = "2024-01-01T00:00:00"
+ self.queried_dttm = "2024-01-01T00:00:00"
+ self.stacktrace = None
+ self.error_message = None
+ self.is_cached = True
+ self.sql_rowcount = 0
+ self.cache_value = None
+ self.applied_template_filters = []
+ self.rejected_filter_columns = []
+ self.annotation_data = {"stale": "should not be used"}
+ self.bq_memory_limited = False
+ self.bq_memory_limited_row_count = 0
+ self.result_persisted = False
+ self.set_query_result = MagicMock()
+
+ mock_cache = MockCache()
+
+ with (
+ patch(
+ "superset.common.query_context_processor.QueryCacheManager"
+ ) as mock_cache_manager,
+ patch.object(query_obj, "validate", return_value=None),
+ patch.object(processor, "query_cache_key", return_value="df-key"),
+ patch.object(processor, "annotation_cache_key", return_value="ann-key"),
+ patch.object(
+ processor, "_get_annotation_data_cached", return_value={"a": [1, 2]}
+ ) as mock_get_annotation,
+ patch.object(processor, "get_cache_timeout", return_value=3600),
+ ):
+ mock_cache_manager.get.return_value = mock_cache
+ result = processor.get_df_payload(query_obj, force_cached=False)
+
+ # The dataframe cache is a hit, so the (expensive) query is never re-run,
+ # and its cache entry is never rewritten.
+ mock_cache.set_query_result.assert_not_called()
+
+ # Annotation data is resolved through the separate, per-user cache path,
+ # keyed by the dedicated annotation cache key -- not the dataframe's key.
+ mock_get_annotation.assert_called_once()
+ _, kwargs = mock_get_annotation.call_args
+ assert kwargs["cache_key"] == "ann-key"
+
+ # The payload serves the freshly-resolved annotation data, not whatever
+ # (stale) value happened to sit on the dataframe's cache object.
+ assert result["annotation_data"] == {"a": [1, 2]}
+
+
+def test_get_df_payload_result_annotation_refresh_independent_of_df_marker():
Review Comment:
<div>
<div id="suggestion">
<div id="issue"><b>Missing type annotations</b></div>
<div id="fix">
BITO.md adaptive rules 7819/12587/12787 require explicit type annotations on new test code, and 12147 requires docstrings on helpers. This test lacks `-> None`; nested `marker_lookup` lacks a return hint and docstring; `MockCache.__init__` and the mock variables lack annotations. Adding them satisfies the org typing standard for new Python code.
</div>
</div>
<small><i>Code Review Run #df4733</i></small>
</div>
---
Should Bito avoid suggestions like this for future reviews? (<a href=https://alpha.bito.ai/home/ai-agents/review-rules>Manage Rules</a>)
- [ ] Yes, avoid them
##########
tests/unit_tests/common/test_query_context_processor.py:
##########
@@ -2226,6 +2396,180 @@ def test_get_df_payload_no_warning_when_not_memory_limited() -> None:
assert result["warning"] is None
+def test_get_df_payload_result_decouples_annotation_cache_from_dataframe_cache():
Review Comment:
<div>
<div id="suggestion">
<div id="issue"><b>Missing Return Type Hint</b></div>
<div id="fix">
BITO.md adaptive rule 7819 mandates explicit return type hints on all functions, explicitly including test methods. This new test omits `-> None:`, while sibling `test_get_df_payload_bq_memory_limited_warning` (line 2283) already annotates it. Adding the one-line hint keeps the mandated standard uniform and enables static type checking.
</div>
</div>
<small><i>Code Review Run #df4733</i></small>
</div>
---
Should Bito avoid suggestions like this for future reviews? (<a href=https://alpha.bito.ai/home/ai-agents/review-rules>Manage Rules</a>)
- [ ] Yes, avoid them
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] fix(charts): decouple annotation cache from the shared dataframe cache [superset]
Posted by "bito-code-review[bot] (via GitHub)" <gi...@apache.org>.
bito-code-review[bot] commented on PR #44406:
URL: https://github.com/apache/superset/pull/44406#issuecomment-5926212786
AI Code Review is in progress (usually takes 3 to 15 minutes unless it's a very large PR).
<details>
<summary>
Bito Usage Guide
</summary>
**Commands**
Type the following command in the pull request comment and save the comment.
- `/review` - Manually triggers an incremental AI Review.
- `/review full` - Manually triggers a full AI Review.
- `/pause` - Pauses automatic reviews on this pull request.
- `/resume` - Resumes automatic reviews.
- `/resolve` - Marks all Bito-posted review comments as resolved.
- `/abort` - Cancels all in-progress reviews.
Refer to the <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/available-commands" target="_blank" rel="noopener noreferrer">documentation</a> for additional commands.
**Configuration**
This repository uses `Superset` You can customize the agent settings <a href="https://alpha.bito.ai/home/ai-agents/code-review-agent" target="_blank" rel="noopener noreferrer">here</a> or contact your Bito workspace admin at evan@preset.io.
**Documentation & Help**
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/getting-started/install-run-using-bito-cloud/create-or-customize-an-agent-instance" target="_blank" rel="noopener noreferrer">Customize agent settings</a>
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/implementing-custom-code-review-rules" target="_blank" rel="noopener noreferrer">Review rules</a>
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent" target="_blank" rel="noopener noreferrer">General documentation</a>
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/faqs" target="_blank" rel="noopener noreferrer">FAQ</a>
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] fix(charts): decouple annotation cache from the shared dataframe cache [superset]
Posted by "sadpandajoe (via GitHub)" <gi...@apache.org>.
sadpandajoe commented on code in PR #44406:
URL: https://github.com/apache/superset/pull/44406#discussion_r4243433681
##########
superset/common/query_context_processor.py:
##########
@@ -72,7 +74,6 @@
get_column_name,
get_column_names_from_columns,
get_column_names_from_metrics,
- get_user_id,
is_adhoc_column,
Review Comment:
Dropping `get_user_id` here leaves the rest of the old annotation-context imports orphaned: `DatasourceDAO`, `DatasourceNotFound`, `DatasourceTypeNotSupportedError`, `DatasourceValueIsIncorrect`, `json` and the `TYPE_CHECKING` `Datasource` no longer have any reference in this file now that `_annotation_source_scope` resolves the datasource through the chart's query context. Ruff's F401 will fail lint/pre-commit on the changed file. Can these be removed in the same pass?
##########
superset/common/query_context_processor.py:
##########
@@ -343,6 +355,32 @@ def get_df_payload_result(
# nonce reads the freshly-cached result instead of recomputing it.
self._mark_force_executed(query_obj, cache_key, cache.result_persisted)
+ # Annotation data is fetched per requesting user (and, for chart-backed
+ # layers, scoped by the referenced chart datasource's RLS), so it is
+ # resolved and cached under its own entry β independent of whether the
+ # (shareable) dataframe above was a hit or a miss β rather than forcing
+ # every viewer of the same chart onto their own full dataframe copy.
+ annotation_data: dict[str, Any] = {}
+ if (
+ query_obj
+ and query_obj.annotation_layers
+ and cache.status != QueryStatus.FAILED
Review Comment:
The caller-level wiring around this block has no direct coverage: every new `get_df_payload_result` test either supplies a successful dataframe cache or replaces `_get_annotation_data_cached` with a mock. If someone drops the `cache.status != QueryStatus.FAILED` guard, a failed dataframe would still trigger the annotation fetch, and an annotation exception could replace the original failure response; likewise nothing asserts the annotation nonce marker is written only when `set_and_log_cache` actually persisted (the existing marker tests call `_mark_force_executed` directly). Could a test drive `get_df_payload_result` with annotation layers through the real helper for both a failed dataframe (annotation helper not called, `annotation_data == {}`, original error kept) and a non-persisting annotation write (no marker, next same-nonce read still forces)?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] fix(charts): decouple annotation cache from the shared dataframe cache [superset]
Posted by "sadpandajoe (via GitHub)" <gi...@apache.org>.
sadpandajoe commented on code in PR #44406:
URL: https://github.com/apache/superset/pull/44406#discussion_r4270481635
##########
tests/unit_tests/common/test_query_context_processor.py:
##########
@@ -2639,6 +3215,101 @@ def test_mark_force_executed_noop_without_nonce(processor, mock_query_context):
cache_manager.data_cache.set.assert_not_called()
+# =============================================================================
+# Annotation-data cache decoupled from the dataframe cache
+# =============================================================================
+
+
+def test_get_annotation_data_cached_reads_from_cache(
+ processor: QueryContextProcessor,
+) -> None:
+ """A hit on the annotation-specific key skips recomputation entirely."""
+ with patch(
+ "superset.common.query_context_processor.cache_manager"
+ ) as cache_manager:
+ cache_manager.data_cache.get.return_value = {"annotation_data": {"a": 1}}
+ with patch.object(processor, "get_annotation_data") as mock_get:
+ result = processor._get_annotation_data_cached(
+ query_obj=MagicMock(),
+ cache_key="ak",
+ force_query=False,
+ force_cached=False,
+ timeout=60,
+ datasource_uid="ds",
+ )
+ assert result == {"a": 1}
+ mock_get.assert_not_called()
+
+
+def test_get_annotation_data_cached_computes_and_caches_on_miss(
+ processor: QueryContextProcessor,
+) -> None:
+ """A miss computes the annotation data and stores it under the scoped key."""
+ with (
+ patch("superset.common.query_context_processor.cache_manager") as cache_manager,
+ patch("superset.common.query_context_processor.set_and_log_cache") as mock_set,
+ ):
+ cache_manager.data_cache.get.return_value = None
+ with patch.object(
+ processor, "get_annotation_data", return_value={"a": 1}
+ ) as mock_get:
+ result = processor._get_annotation_data_cached(
+ query_obj=MagicMock(),
+ cache_key="ak",
+ force_query=False,
+ force_cached=False,
+ timeout=60,
+ datasource_uid="ds",
+ )
+ assert result == {"a": 1}
+ mock_get.assert_called_once()
+ mock_set.assert_called_once_with(
+ cache_manager.data_cache, "ak", {"annotation_data": {"a": 1}}, 60, "ds"
+ )
+
+
+def test_get_annotation_data_cached_force_cached_raises_on_miss(
+ processor: QueryContextProcessor,
+) -> None:
+ """``force_cached`` must never fall through to a live compute -- the same
+ contract ``QueryCacheManager.get`` enforces for the dataframe cache."""
+ with patch(
+ "superset.common.query_context_processor.cache_manager"
+ ) as cache_manager:
+ cache_manager.data_cache.get.return_value = None
+ with pytest.raises(CacheLoadError):
+ processor._get_annotation_data_cached(
+ query_obj=MagicMock(),
+ cache_key="ak",
+ force_query=False,
+ force_cached=True,
+ timeout=60,
+ datasource_uid="ds",
+ )
+
+
+def test_get_annotation_data_cached_force_query_bypasses_cache_read(processor):
+ """A forced refresh recomputes rather than serving a stale cached value."""
+ with (
+ patch("superset.common.query_context_processor.cache_manager") as cache_manager,
+ patch("superset.common.query_context_processor.set_and_log_cache"),
Review Comment:
`set_and_log_cache` is patched with a bare `MagicMock` here and nothing asserts the nonce-marker write that depends on its return value, so replacing `persisted` with `True` at the `_mark_force_executed` call in `_get_annotation_data_cached` would still pass. A forced refresh whose write fails would then record the marker and a same-nonce read-back would serve the stale annotation entry. Could a test at this boundary drive `set_and_log_cache` to return both `True` and `False` and assert the marker is written under the annotation key only on success?
##########
superset/common/query_context_processor.py:
##########
@@ -474,73 +503,226 @@ def query_cache_key(self, query_obj: QueryObject, **kwargs: Any) -> str | None:
)
return cache_key
- def _annotation_cache_context(self, query_obj: QueryObject) -> dict[str, Any]:
+ def annotation_cache_key(self, query_obj: QueryObject) -> str | None:
"""
- Cache-key material binding cached annotation data to its security
- context.
-
- Annotation payloads are fetched per requesting user and stored on the
- same cache entry as the dataframe, so the key also binds the requesting
- user and, for chart-backed layers, the RLS clauses of the referenced
- chart's datasource.
+ Cache key for this query's annotation-layer payload, or ``None`` when
+ the query has no annotation layers or when some layer's access scope
+ could not be resolved (the payload is then never cached, see
+ :meth:`_annotation_source_scope`).
+
+ Annotation payloads are fetched under the requesting user's access
+ scope, which is a stricter security requirement than the dataframe
+ itself has. Keying them separately from :meth:`query_cache_key` keeps
+ that scoping from forcing every distinct viewer of an annotated chart
+ onto their own full copy of the (potentially much larger) shared
+ dataframe: users with the same access scope share this key too.
"""
- source_rls: dict[str, list[str] | None] = {}
- source_versions: dict[str, tuple[str, str]] = {}
- metadata_versions: dict[str, str] = {}
+ if not query_obj or not query_obj.annotation_layers:
+ return None
+ if (annotation_context := self._annotation_cache_context(query_obj)) is None:
+ return None
+ return self.query_cache_key(query_obj, annotation_context=annotation_context)
+
+ def _annotation_cache_context(
+ self, query_obj: QueryObject
+ ) -> dict[str, Any] | None:
+ """
+ Cache-key material binding annotation data to its security *scope* so
+ users with the same access share a cache entry and users with a
+ different scope β or no access β never read each other's data.
+
+ * NATIVE layers: the ``can_read`` permission on ``Annotation``, the
+ only user-dependent dimension of these global records.
+ * Chart-backed (``line``/``table``) layers: see
+ :meth:`_annotation_source_scope`, recorded per *layer* (not per
+ referenced chart) since two layers can reference the same chart
+ with different overrides, and so different scopes.
+
+ Returns ``None`` when any layer's scope cannot be resolved, meaning
+ the payload must not be cached at all.
+ """
+ context: dict[str, Any] = {}
+
+ if any(
+ layer.get("sourceType") == "NATIVE" for layer in query_obj.annotation_layers
+ ):
+ context["annotation_read"] = security_manager.can_access(
+ "can_read", "Annotation"
+ )
+
+ source_scope: list[dict[str, Any]] = []
for layer in query_obj.annotation_layers:
if (
layer.get("sourceType")
not in ANNOTATION_SOURCE_TYPES_WITH_CHART_REFERENCE
):
continue
- layer_value = layer.get("value")
+ if (scope := self._annotation_source_scope(layer)) is None:
+ return None
+ source_scope.append({"value": str(layer.get("value")), **scope})
+ if source_scope:
+ context["source_scope"] = source_scope
+
+ return context
+
+ def _annotation_source_scope(self, layer: dict[str, Any]) -> dict[str, Any] | None:
+ """
+ Access and data-identity cache-key material for one chart-backed
+ annotation layer, or ``None`` when it cannot be resolved.
+
+ ``None`` means "do not cache": a missing chart, a chart with neither
+ a saved query context nor a resolvable datasource, or any error while
+ deriving the scope would otherwise collapse an authorized and a
+ denied requester onto one shared identity, letting the latter read
+ the former's warmed payload. The caller fetches live instead.
+
+ ``access`` keeps a user denied the referenced chart's datasource from
+ reading an authorized user's cached payload. When the chart has a
+ saved query context, this runs the *same* authorization path
+ :meth:`get_viz_annotation_data` executes
+ (``QueryContext.raise_for_access``) rather than the coarser
+ :meth:`SecurityManager.can_access_datasource` β a requester whose
+ access comes from a dashboard/viewer-promiscuous-mode bypass (which
+ depends on the chart's own saved ``form_data``, e.g. its
+ ``slice_id``/``dashboardId``) would otherwise still fail that coarser,
+ context-free check and collapse onto the same denied scope as a
+ genuinely unauthorized requester, letting the latter read the
+ former's cached payload. ``data_key`` is the annotation chart's own
+ query cache key(s) β derived from the same, override-applied query
+ objects actually executed (see :meth:`_apply_annotation_overrides`),
+ so it captures the datasource version, RLS clauses, and any per-user
+ Jinja/virtual-dataset RLS material exactly as the live fetch would,
+ including material an override only introduces at a finer grain.
+ Reusing this logic avoids re-deriving it and automatically inherits
+ any future correctness fixes made there.
+ """
+ layer_value = layer.get("value")
+ try:
chart = (
ChartDAO.find_by_id(layer_value) if layer_value is not None else None
)
- annotation_datasource: Datasource | None = (
- chart.resolved_datasource if chart else None
- )
- if chart and isinstance(chart.query_context, str):
+ if chart is None:
+ return None
+
+ # The saved query context is what the live fetch executes and
+ # authorizes against, so it is consulted first, independent of
+ # the chart's own datasource pointer: a chart can have its
+ # ``datasource_id`` cleared while its saved query context still
+ # targets a valid dataset, and the live fetch would still work.
+ annotation_query_context = chart.get_query_context()
+ if annotation_query_context is not None:
+ self._apply_annotation_overrides(annotation_query_context, layer)
try:
- source: dict[str, Any] = json.loads(chart.query_context)[
- "datasource"
- ]
- annotation_datasource = DatasourceDAO.get_datasource(
- source["type"], source["id"]
- )
- except (
- ValueError,
- TypeError,
- KeyError,
- DatasourceNotFound,
- DatasourceTypeNotSupportedError,
- DatasourceValueIsIncorrect,
- ):
- annotation_datasource = None
+ annotation_query_context.raise_for_access()
+ access: Any = True
+ except SupersetSecurityException:
+ access = False
+ data_key: Any = [
+ annotation_query_context.query_cache_key(query_object)
Review Comment:
With `ALLOW_ADHOC_SUBQUERY` on, an annotation-source chart whose metric is a SQL expression with a subquery (`MAX((SELECT SUM(amount) FROM sales))`) gets its tenant RLS on `sales` injected at execution time, but that RLS is not part of `query_cache_key` or `get_rls_cache_key` here. Two users with the same dataset grant and different RLS on `sales` therefore get identical `access` and `data_key`, and when the source chart does not cache itself (`cache_timeout=-1`) the second user is served the first user's warmed total. The old `user_id` binding isolated this. Should the scope include the RLS predicates of tables referenced inside adhoc expressions, or skip caching when an adhoc subquery is present?
##########
tests/unit_tests/common/test_query_context_processor.py:
##########
@@ -3115,11 +3788,17 @@ def payload(context: QueryContext, **kwargs: Any) -> dict[str, Any]:
"superset.common.query_context_processor.security_manager.get_rls_cache_key",
return_value=[],
),
+ patch.object(QueryContext, "raise_for_access"),
+ patch.object(QueryContext, "query_cache_key", return_value="data-key"),
+ patch(
+ "superset.common.query_context_processor.security_manager"
+ ".can_access_datasource",
+ return_value=True,
+ ),
patch(
"superset.common.query_context_processor.ChartDAO.find_by_id",
return_value=chart,
),
- patch("superset.common.query_context_processor.get_user_id", return_value=42),
):
context: dict[str, Any] = processor._annotation_cache_context(query)
Review Comment:
`_annotation_cache_context` now returns `dict[str, Any] | None`, but the result is assigned to a `dict[str, Any]` local in a function that is fully annotated (`-> None`), so mypy checks it and will report an incompatible assignment. Could this declare the optional type and assert it is not `None` before indexing `context["source_scope"]`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] fix(charts): decouple annotation cache from the shared dataframe cache [superset]
Posted by "sadpandajoe (via GitHub)" <gi...@apache.org>.
sadpandajoe commented on code in PR #44406:
URL: https://github.com/apache/superset/pull/44406#discussion_r4345616115
##########
tests/unit_tests/tasks/test_query_cancel.py:
##########
@@ -159,6 +159,51 @@ def test_capture_registers_abort_handler_once_when_id_available() -> None:
assert ordered.index("set_cancellation") < ordered.index("on_abort")
+def test_capture_ignores_cursor_from_a_different_database() -> None:
+ from superset.tasks.async_queries import _capture_query_cancellation
+
+ ctx = MagicMock()
+ qc = _query_context("42")
+ other_database = MagicMock()
+ other_database.id = qc.datasource.database.id + 1
+ with (
+ patch("superset.tasks.async_queries.get_context", return_value=ctx),
+ patch("superset.tasks.async_queries.current_app"),
+ ):
+ with _capture_query_cancellation(qc):
+ sink = _cancel_id_sink.get()
+ assert sink is not None
+ sink(MagicMock(), other_database)
+
+ ctx.on_abort.assert_not_called()
+ ctx.set_cancellation.assert_not_called()
+
+
+def test_notify_cursor_routes_by_database_through_the_real_seam() -> None:
+ """Driven through ``notify_cursor`` (the seam ``get_raw_connection`` calls)
+ rather than the sink directly: the task's own database registers the abort
+ handler exactly once, another database's cursor registers nothing."""
+ from superset.tasks.async_queries import _capture_query_cancellation
+
+ ctx = MagicMock()
+ qc = _query_context("42")
+ own_database = qc.datasource.database
+ other_database = MagicMock()
+ other_database.id = own_database.id + 1
+ with (
+ patch("superset.tasks.async_queries.get_context", return_value=ctx),
+ patch("superset.tasks.async_queries.current_app"),
+ ):
+ with _capture_query_cancellation(qc):
+ notify_cursor(MagicMock(), other_database)
Review Comment:
This still bypasses the production caller: `notify_cursor` is invoked directly with the database, so reverting `Database._execute_sql_with_mutation_and_logging` in `models/core.py` back to `notify_cursor(cursor)` keeps every test green. In that case a warm host dataframe with a cold annotation layer on another PostgreSQL server records the annotation cursor against the host database, and abort or timeout can cancel the wrong server's session. Could a test run the real execute path under `capture_cancel_id` (or assert the `notify_cursor` call args from it) and check the host database registers the handle while another database does not?
##########
superset/common/query_context_processor.py:
##########
@@ -478,73 +518,261 @@ def query_cache_key(self, query_obj: QueryObject, **kwargs: Any) -> str | None:
)
return cache_key
- def _annotation_cache_context(self, query_obj: QueryObject) -> dict[str, Any]:
+ def annotation_cache_key(self, query_obj: QueryObject) -> str | None:
"""
- Cache-key material binding cached annotation data to its security
- context.
-
- Annotation payloads are fetched per requesting user and stored on the
- same cache entry as the dataframe, so the key also binds the requesting
- user and, for chart-backed layers, the RLS clauses of the referenced
- chart's datasource.
+ Cache key for this query's annotation-layer payload, or ``None`` when
+ the query has no annotation layers or when some layer's access scope
+ could not be resolved (the payload is then never cached, see
+ :meth:`_annotation_source_scope`).
+
+ Annotation payloads are fetched under the requesting user's access
+ scope, which is a stricter security requirement than the dataframe
+ itself has. Keying them separately from :meth:`query_cache_key` keeps
+ that scoping from forcing every distinct viewer of an annotated chart
+ onto their own full copy of the (potentially much larger) shared
+ dataframe: users with the same access scope share this key too.
+ """
+ if not query_obj or not query_obj.annotation_layers:
+ return None
+ if (annotation_context := self._annotation_cache_context(query_obj)) is None:
+ return None
+ return self.query_cache_key(query_obj, annotation_context=annotation_context)
+
+ def _annotation_cache_context(
+ self, query_obj: QueryObject
+ ) -> dict[str, Any] | None:
+ """
+ Cache-key material binding annotation data to its security *scope* so
+ users with the same access share a cache entry and users with a
+ different scope β or no access β never read each other's data.
+
+ * NATIVE layers: the ``can_read`` permission on ``Annotation``, the
+ only user-dependent dimension of these global records.
+ * Chart-backed (``line``/``table``) layers: see
+ :meth:`_annotation_source_scope`, recorded per *layer* (not per
+ referenced chart) since two layers can reference the same chart
+ with different overrides, and so different scopes.
+
+ Returns ``None`` when any layer's scope cannot be resolved, meaning
+ the payload must not be cached at all.
"""
- source_rls: dict[str, list[str] | None] = {}
- source_versions: dict[str, tuple[str, str]] = {}
- metadata_versions: dict[str, str] = {}
+ context: dict[str, Any] = {}
+
+ if any(
+ layer.get("sourceType") == "NATIVE" for layer in query_obj.annotation_layers
+ ):
+ context["annotation_read"] = security_manager.can_access(
+ "can_read", "Annotation"
+ )
+
+ source_scope: list[dict[str, Any]] = []
for layer in query_obj.annotation_layers:
if (
layer.get("sourceType")
not in ANNOTATION_SOURCE_TYPES_WITH_CHART_REFERENCE
):
continue
- layer_value = layer.get("value")
+ if (scope := self._annotation_source_scope(layer)) is None:
+ return None
+ source_scope.append({"value": str(layer.get("value")), **scope})
+ if source_scope:
+ context["source_scope"] = source_scope
+
+ return context
+
+ def _annotation_source_scope(self, layer: dict[str, Any]) -> dict[str, Any] | None:
+ """
+ Access and data-identity cache-key material for one chart-backed
+ annotation layer, or ``None`` when it cannot be resolved.
+
+ ``None`` means "do not cache": a missing chart, a chart with neither
+ a saved query context nor a resolvable datasource, or any error while
+ deriving the scope would otherwise collapse an authorized and a
+ denied requester onto one shared identity, letting the latter read
+ the former's warmed payload. The caller fetches live instead. A
+ referenced chart whose own caching is disabled
+ (``CACHE_DISABLED_TIMEOUT``) is ``None`` too: its rows are never
+ shared through its own cache, so they must not be shared through the
+ host chart's annotation entry either.
+
+ ``access`` keeps a user denied the referenced chart's datasource from
+ reading an authorized user's cached payload. When the chart has a
+ saved query context, this runs the *same* authorization path
+ :meth:`get_viz_annotation_data` executes
+ (``QueryContext.raise_for_access``) rather than the coarser
+ :meth:`SecurityManager.can_access_datasource` β a requester whose
+ access comes from a dashboard/viewer-promiscuous-mode bypass (which
+ depends on the chart's own saved ``form_data``, e.g. its
+ ``slice_id``/``dashboardId``) would otherwise still fail that coarser,
+ context-free check and collapse onto the same denied scope as a
+ genuinely unauthorized requester, letting the latter read the
+ former's cached payload. ``data_key`` is the annotation chart's own
+ query cache key(s) β derived from the same, override-applied and
+ result-type-prepared query objects actually executed (see
+ :meth:`_apply_annotation_overrides` and
+ :func:`~superset.common.query_actions.prepare_query_object`), so it
+ captures the datasource version, RLS clauses, and any per-user
+ Jinja/virtual-dataset RLS material exactly as the live fetch would,
+ including material an override only introduces at a finer grain or
+ that a samples query's cleared metrics only expose at execution.
+ Reusing this logic avoids re-deriving it and automatically inherits
+ any future correctness fixes made there.
+ """
+ layer_value = layer.get("value")
+ try:
chart = (
ChartDAO.find_by_id(layer_value) if layer_value is not None else None
)
- annotation_datasource: Datasource | None = (
- chart.resolved_datasource if chart else None
- )
- if chart and isinstance(chart.query_context, str):
+ if chart is None:
+ return None
+
+ # The saved query context is what the live fetch executes and
+ # authorizes against, so it is consulted first, independent of
+ # the chart's own datasource pointer: a chart can have its
+ # ``datasource_id`` cleared while its saved query context still
+ # targets a valid dataset, and the live fetch would still work.
+ annotation_query_context = chart.get_query_context()
+ if annotation_query_context is not None:
+ if (
+ annotation_query_context.get_cache_timeout()
+ == CACHE_DISABLED_TIMEOUT
+ ):
+ return None
+ self._apply_annotation_overrides(annotation_query_context, layer)
try:
- source: dict[str, Any] = json.loads(chart.query_context)[
- "datasource"
+ annotation_query_context.raise_for_access()
+ access: Any = True
+ except SupersetSecurityException:
+ access = False
+ # A denied requester never reads or writes a cached payload,
+ # so skip deriving a data identity (which can render the
+ # dataset's Jinja) for a resource they cannot access.
+ # Keyed on the query object the fetch will execute: a samples
+ # or drill-detail result type rewrites the saved query (e.g.
+ # clears its metrics) before running it, and that can change
+ # the per-user Jinja material the key has to carry.
+ data_key: Any = (
+ [
+ annotation_query_context.query_cache_key(
Review Comment:
Whitespace-separated identity macros still escape this key: `QueryContext.query_cache_key` only renders the dataset's Jinja when `SqlaTable.has_extra_cache_key_calls` matches `ExtraCache.regex`, and that regex requires `current_user_id(` with no space. With an RLS clause like `owner_id = {{ current_user_id () }}` on the annotation source's dataset, two users with the same grants derive the same `data_key` (the raw RLS clause is identical and no user material is added), so the second viewer is served the first viewer's cached annotation rows. Before this PR the annotation context bound the requester's user ID, which hid that gap. Should an annotation entry only be shared when the key demonstrably carries the user-dependent material, rather than relying on that syntactic gate?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] fix(charts): decouple annotation cache from the shared dataframe cache [superset]
Posted by "bito-code-review[bot] (via GitHub)" <gi...@apache.org>.
bito-code-review[bot] commented on PR #44406:
URL: https://github.com/apache/superset/pull/44406#issuecomment-5866532013
AI Code Review is in progress (usually takes 3 to 15 minutes unless it's a very large PR).
<details>
<summary>
Bito Usage Guide
</summary>
**Commands**
Type the following command in the pull request comment and save the comment.
- `/review` - Manually triggers an incremental AI Review.
- `/review full` - Manually triggers a full AI Review.
- `/pause` - Pauses automatic reviews on this pull request.
- `/resume` - Resumes automatic reviews.
- `/resolve` - Marks all Bito-posted review comments as resolved.
- `/abort` - Cancels all in-progress reviews.
Refer to the <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/available-commands" target="_blank" rel="noopener noreferrer">documentation</a> for additional commands.
**Configuration**
This repository uses `Superset` You can customize the agent settings <a href="https://alpha.bito.ai/home/ai-agents/code-review-agent" target="_blank" rel="noopener noreferrer">here</a> or contact your Bito workspace admin at evan@preset.io.
**Documentation & Help**
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/getting-started/install-run-using-bito-cloud/create-or-customize-an-agent-instance" target="_blank" rel="noopener noreferrer">Customize agent settings</a>
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/implementing-custom-code-review-rules" target="_blank" rel="noopener noreferrer">Review rules</a>
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent" target="_blank" rel="noopener noreferrer">General documentation</a>
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/faqs" target="_blank" rel="noopener noreferrer">FAQ</a>
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] fix(charts): decouple annotation cache from the shared dataframe cache [superset]
Posted by "bito-code-review[bot] (via GitHub)" <gi...@apache.org>.
bito-code-review[bot] commented on PR #44406:
URL: https://github.com/apache/superset/pull/44406#issuecomment-5722232175
AI Code Review is in progress (usually takes 3 to 15 minutes unless it's a very large PR).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] fix(charts): decouple annotation cache from the shared dataframe cache [superset]
Posted by "bito-code-review[bot] (via GitHub)" <gi...@apache.org>.
bito-code-review[bot] commented on PR #44406:
URL: https://github.com/apache/superset/pull/44406#issuecomment-5729838659
AI Code Review is in progress (usually takes 3 to 15 minutes unless it's a very large PR).
<details>
<summary>
Bito Usage Guide
</summary>
**Commands**
Type the following command in the pull request comment and save the comment.
- `/review` - Manually triggers an incremental AI Review.
- `/review full` - Manually triggers a full AI Review.
- `/pause` - Pauses automatic reviews on this pull request.
- `/resume` - Resumes automatic reviews.
- `/resolve` - Marks all Bito-posted review comments as resolved.
- `/abort` - Cancels all in-progress reviews.
Refer to the <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/available-commands" target="_blank" rel="noopener noreferrer">documentation</a> for additional commands.
**Configuration**
This repository uses `Superset` You can customize the agent settings <a href="https://alpha.bito.ai/home/ai-agents/code-review-agent" target="_blank" rel="noopener noreferrer">here</a> or contact your Bito workspace admin at evan@preset.io.
**Documentation & Help**
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/getting-started/install-run-using-bito-cloud/create-or-customize-an-agent-instance" target="_blank" rel="noopener noreferrer">Customize agent settings</a>
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/implementing-custom-code-review-rules" target="_blank" rel="noopener noreferrer">Review rules</a>
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent" target="_blank" rel="noopener noreferrer">General documentation</a>
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/faqs" target="_blank" rel="noopener noreferrer">FAQ</a>
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] fix(charts): decouple annotation cache from the shared dataframe cache [superset]
Posted by "bito-code-review[bot] (via GitHub)" <gi...@apache.org>.
bito-code-review[bot] commented on code in PR #44406:
URL: https://github.com/apache/superset/pull/44406#discussion_r4170191341
##########
tests/unit_tests/common/test_query_context_processor.py:
##########
@@ -117,25 +118,259 @@ def processor(mock_query_context):
return processor
-def test_query_cache_key_binds_annotation_data_to_requesting_user(processor):
- """The cache key for annotated queries must differ per requesting user."""
+def test_annotation_cache_key_binds_native_annotation_read_scope(processor) -> None:
+ """The annotation cache key for NATIVE layers must differ when the
+ requester's ``can_read`` (Annotation) access differs -- not who they are."""
query_obj = MagicMock()
query_obj.annotation_layers = [{"sourceType": "NATIVE", "name": "a", "value": 1}]
- with (
- patch(
- "superset.common.query_context_processor.get_user_id",
- side_effect=[1, 2],
- ),
- patch("superset.common.query_context_processor.security_manager"),
- ):
- processor.query_cache_key(query_obj)
- processor.query_cache_key(query_obj)
+ # ``security_manager`` autodetects as an async spec under a bare
+ # ``patch()`` (its real object trips ``unittest.mock``'s coroutine
+ # inference), which would silently turn every attribute access into an
+ # ``AsyncMock`` returning a fresh unawaited coroutine per call -- always
+ # unequal to itself and never equal to a configured return value. Forcing
+ # ``new_callable=MagicMock`` keeps these synchronous, as the real object
+ # is.
+ with patch(
+ "superset.common.query_context_processor.security_manager",
+ new_callable=MagicMock,
+ ) as security_manager:
+ security_manager.can_access.side_effect = [True, False]
+ processor.annotation_cache_key(query_obj)
+ processor.annotation_cache_key(query_obj)
contexts = [
call.kwargs["annotation_context"] for call in query_obj.cache_key.call_args_list
]
assert contexts[0] != contexts[1]
+def test_annotation_cache_key_shares_across_same_access_scope() -> None:
+ """Two distinct requesters (separate processor/query-object instances,
+ standing in for two different requests) with identical access scope must
+ produce identical annotation-context material. Reusing a single
+ processor/query_obj across both calls (as this test previously did)
+ would pass trivially regardless of whether the key is scope-based or
+ identity-based, since nothing about "who's asking" would ever vary."""
+ layer = {"sourceType": "NATIVE", "name": "a", "value": 1}
+ processor_a = QueryContextProcessor(MagicMock())
+ processor_b = QueryContextProcessor(MagicMock())
+ query_obj_a = MagicMock(annotation_layers=[layer])
+ query_obj_b = MagicMock(annotation_layers=[layer])
+
+ with patch(
+ "superset.common.query_context_processor.security_manager",
+ new_callable=MagicMock,
+ ) as security_manager:
+ security_manager.can_access.return_value = True
+ context_a = processor_a._annotation_cache_context(query_obj_a)
+ context_b = processor_b._annotation_cache_context(query_obj_b)
+
+ assert context_a == context_b
+
+
+def test_query_cache_key_does_not_bind_annotation_scope(processor) -> None:
+ """The dataframe cache key must stay shared across viewers of the same
+ chart, even when the query has annotation layers β only the separate
+ annotation cache key (see above) carries access-scope material."""
+ query_obj = MagicMock()
+ query_obj.annotation_layers = [{"sourceType": "NATIVE", "name": "a", "value": 1}]
+ with patch(
+ "superset.common.query_context_processor.security_manager",
+ new_callable=MagicMock,
+ ):
+ processor.query_cache_key(query_obj)
+ processor.query_cache_key(query_obj)
+ for call in query_obj.cache_key.call_args_list:
+ assert "annotation_context" not in call.kwargs
+
+
+@pytest.fixture
+def mock_annotation_chart() -> Iterator[MagicMock]:
+ """A found chart, wired as the referenced chart for
+ ``_annotation_source_scope`` tests -- factors out the repeated
+ ``ChartDAO.find_by_id`` patch those tests all need."""
+ chart = MagicMock()
+ with patch(
+ "superset.common.query_context_processor.ChartDAO.find_by_id",
+ return_value=chart,
+ ):
+ yield chart
+
+
+def test_annotation_source_scope_binds_datasource_access(
+ processor, mock_annotation_chart
+) -> None:
+ """A chart-backed annotation layer's scope must differ when the
+ requester's access to the referenced datasource differs."""
+ mock_annotation_chart.get_query_context.return_value = None
+ with patch(
+ "superset.common.query_context_processor.security_manager",
+ new_callable=MagicMock,
+ ) as security_manager:
+ security_manager.can_access_datasource.side_effect = [True, False]
+ security_manager.get_rls_cache_key.return_value = []
+ scope_a = processor._annotation_source_scope({"value": 1})
+ scope_b = processor._annotation_source_scope({"value": 1})
+ assert scope_a != scope_b
+ assert scope_a["access"] is True
+ assert scope_b["access"] is False
+
+
+def test_annotation_source_scope_reuses_referenced_chart_cache_key(
+ processor, mock_annotation_chart
+) -> None:
+ """When the referenced chart has a saved query context, its own cache
+ key(s) -- covering RLS and per-user Jinja/virtual-dataset material -- are
+ reused rather than re-derived."""
+ mock_query_object = MagicMock()
+ mock_query_context = MagicMock()
+ mock_query_context.queries = [mock_query_object]
+ mock_query_context.query_cache_key.return_value = "referenced-chart-key"
+ mock_annotation_chart.get_query_context.return_value = mock_query_context
+ with patch(
+ "superset.common.query_context_processor.security_manager",
+ new_callable=MagicMock,
+ ) as security_manager:
+ security_manager.can_access_datasource.return_value = True
+ scope = processor._annotation_source_scope({"value": 1})
+ assert scope == {"access": True, "data_key": ["referenced-chart-key"]}
+ mock_query_context.query_cache_key.assert_called_once_with(mock_query_object)
+
+
+def test_annotation_source_scope_uses_live_fetch_authorization(
+ processor, mock_annotation_chart
+) -> None:
Review Comment:
<div>
<div id="suggestion">
<div id="issue"><b>Fixture params unannotated</b></div>
<div id="fix">
Same BITO 11810/12490 violation as the first test (line 218): the fixture-injected parameters `processor` and `mock_annotation_chart` in this new test function lack explicit type annotations. Both types are already imported in this module.
</div>
</div>
<small><i>Code Review Run #a2d0a6</i></small>
</div>
---
Should Bito avoid suggestions like this for future reviews? (<a href=https://alpha.bito.ai/home/ai-agents/review-rules>Manage Rules</a>)
- [ ] Yes, avoid them
##########
tests/unit_tests/common/test_query_context_processor.py:
##########
@@ -117,25 +118,259 @@ def processor(mock_query_context):
return processor
-def test_query_cache_key_binds_annotation_data_to_requesting_user(processor):
- """The cache key for annotated queries must differ per requesting user."""
+def test_annotation_cache_key_binds_native_annotation_read_scope(processor) -> None:
+ """The annotation cache key for NATIVE layers must differ when the
+ requester's ``can_read`` (Annotation) access differs -- not who they are."""
query_obj = MagicMock()
query_obj.annotation_layers = [{"sourceType": "NATIVE", "name": "a", "value": 1}]
- with (
- patch(
- "superset.common.query_context_processor.get_user_id",
- side_effect=[1, 2],
- ),
- patch("superset.common.query_context_processor.security_manager"),
- ):
- processor.query_cache_key(query_obj)
- processor.query_cache_key(query_obj)
+ # ``security_manager`` autodetects as an async spec under a bare
+ # ``patch()`` (its real object trips ``unittest.mock``'s coroutine
+ # inference), which would silently turn every attribute access into an
+ # ``AsyncMock`` returning a fresh unawaited coroutine per call -- always
+ # unequal to itself and never equal to a configured return value. Forcing
+ # ``new_callable=MagicMock`` keeps these synchronous, as the real object
+ # is.
+ with patch(
+ "superset.common.query_context_processor.security_manager",
+ new_callable=MagicMock,
+ ) as security_manager:
+ security_manager.can_access.side_effect = [True, False]
+ processor.annotation_cache_key(query_obj)
+ processor.annotation_cache_key(query_obj)
contexts = [
call.kwargs["annotation_context"] for call in query_obj.cache_key.call_args_list
]
assert contexts[0] != contexts[1]
+def test_annotation_cache_key_shares_across_same_access_scope() -> None:
+ """Two distinct requesters (separate processor/query-object instances,
+ standing in for two different requests) with identical access scope must
+ produce identical annotation-context material. Reusing a single
+ processor/query_obj across both calls (as this test previously did)
+ would pass trivially regardless of whether the key is scope-based or
+ identity-based, since nothing about "who's asking" would ever vary."""
+ layer = {"sourceType": "NATIVE", "name": "a", "value": 1}
+ processor_a = QueryContextProcessor(MagicMock())
+ processor_b = QueryContextProcessor(MagicMock())
+ query_obj_a = MagicMock(annotation_layers=[layer])
+ query_obj_b = MagicMock(annotation_layers=[layer])
+
+ with patch(
+ "superset.common.query_context_processor.security_manager",
+ new_callable=MagicMock,
+ ) as security_manager:
+ security_manager.can_access.return_value = True
+ context_a = processor_a._annotation_cache_context(query_obj_a)
+ context_b = processor_b._annotation_cache_context(query_obj_b)
+
+ assert context_a == context_b
+
+
+def test_query_cache_key_does_not_bind_annotation_scope(processor) -> None:
+ """The dataframe cache key must stay shared across viewers of the same
+ chart, even when the query has annotation layers β only the separate
+ annotation cache key (see above) carries access-scope material."""
+ query_obj = MagicMock()
+ query_obj.annotation_layers = [{"sourceType": "NATIVE", "name": "a", "value": 1}]
+ with patch(
+ "superset.common.query_context_processor.security_manager",
+ new_callable=MagicMock,
+ ):
+ processor.query_cache_key(query_obj)
+ processor.query_cache_key(query_obj)
+ for call in query_obj.cache_key.call_args_list:
+ assert "annotation_context" not in call.kwargs
+
+
+@pytest.fixture
+def mock_annotation_chart() -> Iterator[MagicMock]:
+ """A found chart, wired as the referenced chart for
+ ``_annotation_source_scope`` tests -- factors out the repeated
+ ``ChartDAO.find_by_id`` patch those tests all need."""
+ chart = MagicMock()
+ with patch(
+ "superset.common.query_context_processor.ChartDAO.find_by_id",
+ return_value=chart,
+ ):
+ yield chart
+
+
+def test_annotation_source_scope_binds_datasource_access(
+ processor, mock_annotation_chart
+) -> None:
+ """A chart-backed annotation layer's scope must differ when the
+ requester's access to the referenced datasource differs."""
+ mock_annotation_chart.get_query_context.return_value = None
+ with patch(
+ "superset.common.query_context_processor.security_manager",
+ new_callable=MagicMock,
+ ) as security_manager:
+ security_manager.can_access_datasource.side_effect = [True, False]
+ security_manager.get_rls_cache_key.return_value = []
+ scope_a = processor._annotation_source_scope({"value": 1})
+ scope_b = processor._annotation_source_scope({"value": 1})
+ assert scope_a != scope_b
+ assert scope_a["access"] is True
+ assert scope_b["access"] is False
+
+
+def test_annotation_source_scope_reuses_referenced_chart_cache_key(
+ processor, mock_annotation_chart
+) -> None:
Review Comment:
<div>
<div id="suggestion">
<div id="issue"><b>Fixture params unannotated</b></div>
<div id="fix">
BITO adaptive rules 11810/12490 require explicit type annotations on fixture-injected parameters in new test functions. `processor` and `mock_annotation_chart` are unannotated here (the same pattern repeats at line 239). Both types are already imported in this module, so the annotation is a one-line change per test.
</div>
</div>
<small><i>Code Review Run #a2d0a6</i></small>
</div>
---
Should Bito avoid suggestions like this for future reviews? (<a href=https://alpha.bito.ai/home/ai-agents/review-rules>Manage Rules</a>)
- [ ] Yes, avoid them
##########
tests/unit_tests/common/test_query_context_processor.py:
##########
@@ -2630,6 +3041,94 @@ def test_mark_force_executed_noop_without_nonce(processor, mock_query_context):
cache_manager.data_cache.set.assert_not_called()
+# =============================================================================
+# Annotation-data cache decoupled from the dataframe cache
+# =============================================================================
+
+
+def test_get_annotation_data_cached_reads_from_cache(processor):
Review Comment:
<div>
<div id="suggestion">
<div id="issue"><b>Missing return type hints</b></div>
<div id="fix">
BITO.md adaptive rules 7819/12898 require explicit return type annotations on all new test functions. The three new tests here (lines 3049, 3068, 3092) omit `-> None`, unlike the 12 annotated tests already in this file. Adding `-> None` keeps the file consistent with the mandated standard and lets static checks verify test contracts.
</div>
</div>
<small><i>Code Review Run #a2d0a6</i></small>
</div>
---
Should Bito avoid suggestions like this for future reviews? (<a href=https://alpha.bito.ai/home/ai-agents/review-rules>Manage Rules</a>)
- [ ] Yes, avoid them
##########
tests/unit_tests/common/test_query_context_processor.py:
##########
@@ -117,25 +118,259 @@ def processor(mock_query_context):
return processor
-def test_query_cache_key_binds_annotation_data_to_requesting_user(processor):
- """The cache key for annotated queries must differ per requesting user."""
+def test_annotation_cache_key_binds_native_annotation_read_scope(processor) -> None:
+ """The annotation cache key for NATIVE layers must differ when the
+ requester's ``can_read`` (Annotation) access differs -- not who they are."""
query_obj = MagicMock()
query_obj.annotation_layers = [{"sourceType": "NATIVE", "name": "a", "value": 1}]
- with (
- patch(
- "superset.common.query_context_processor.get_user_id",
- side_effect=[1, 2],
- ),
- patch("superset.common.query_context_processor.security_manager"),
- ):
- processor.query_cache_key(query_obj)
- processor.query_cache_key(query_obj)
+ # ``security_manager`` autodetects as an async spec under a bare
+ # ``patch()`` (its real object trips ``unittest.mock``'s coroutine
+ # inference), which would silently turn every attribute access into an
+ # ``AsyncMock`` returning a fresh unawaited coroutine per call -- always
+ # unequal to itself and never equal to a configured return value. Forcing
+ # ``new_callable=MagicMock`` keeps these synchronous, as the real object
+ # is.
+ with patch(
+ "superset.common.query_context_processor.security_manager",
+ new_callable=MagicMock,
+ ) as security_manager:
+ security_manager.can_access.side_effect = [True, False]
+ processor.annotation_cache_key(query_obj)
+ processor.annotation_cache_key(query_obj)
contexts = [
call.kwargs["annotation_context"] for call in query_obj.cache_key.call_args_list
]
assert contexts[0] != contexts[1]
+def test_annotation_cache_key_shares_across_same_access_scope() -> None:
+ """Two distinct requesters (separate processor/query-object instances,
+ standing in for two different requests) with identical access scope must
+ produce identical annotation-context material. Reusing a single
+ processor/query_obj across both calls (as this test previously did)
+ would pass trivially regardless of whether the key is scope-based or
+ identity-based, since nothing about "who's asking" would ever vary."""
+ layer = {"sourceType": "NATIVE", "name": "a", "value": 1}
+ processor_a = QueryContextProcessor(MagicMock())
+ processor_b = QueryContextProcessor(MagicMock())
+ query_obj_a = MagicMock(annotation_layers=[layer])
+ query_obj_b = MagicMock(annotation_layers=[layer])
+
+ with patch(
+ "superset.common.query_context_processor.security_manager",
+ new_callable=MagicMock,
+ ) as security_manager:
+ security_manager.can_access.return_value = True
+ context_a = processor_a._annotation_cache_context(query_obj_a)
+ context_b = processor_b._annotation_cache_context(query_obj_b)
+
+ assert context_a == context_b
+
+
+def test_query_cache_key_does_not_bind_annotation_scope(processor) -> None:
+ """The dataframe cache key must stay shared across viewers of the same
+ chart, even when the query has annotation layers β only the separate
+ annotation cache key (see above) carries access-scope material."""
+ query_obj = MagicMock()
+ query_obj.annotation_layers = [{"sourceType": "NATIVE", "name": "a", "value": 1}]
+ with patch(
+ "superset.common.query_context_processor.security_manager",
+ new_callable=MagicMock,
+ ):
+ processor.query_cache_key(query_obj)
+ processor.query_cache_key(query_obj)
+ for call in query_obj.cache_key.call_args_list:
+ assert "annotation_context" not in call.kwargs
+
+
+@pytest.fixture
+def mock_annotation_chart() -> Iterator[MagicMock]:
+ """A found chart, wired as the referenced chart for
+ ``_annotation_source_scope`` tests -- factors out the repeated
+ ``ChartDAO.find_by_id`` patch those tests all need."""
+ chart = MagicMock()
+ with patch(
+ "superset.common.query_context_processor.ChartDAO.find_by_id",
+ return_value=chart,
+ ):
+ yield chart
+
+
+def test_annotation_source_scope_binds_datasource_access(
+ processor, mock_annotation_chart
+) -> None:
+ """A chart-backed annotation layer's scope must differ when the
+ requester's access to the referenced datasource differs."""
+ mock_annotation_chart.get_query_context.return_value = None
+ with patch(
+ "superset.common.query_context_processor.security_manager",
+ new_callable=MagicMock,
+ ) as security_manager:
+ security_manager.can_access_datasource.side_effect = [True, False]
+ security_manager.get_rls_cache_key.return_value = []
+ scope_a = processor._annotation_source_scope({"value": 1})
+ scope_b = processor._annotation_source_scope({"value": 1})
+ assert scope_a != scope_b
+ assert scope_a["access"] is True
+ assert scope_b["access"] is False
+
+
+def test_annotation_source_scope_reuses_referenced_chart_cache_key(
+ processor, mock_annotation_chart
+) -> None:
+ """When the referenced chart has a saved query context, its own cache
+ key(s) -- covering RLS and per-user Jinja/virtual-dataset material -- are
+ reused rather than re-derived."""
+ mock_query_object = MagicMock()
+ mock_query_context = MagicMock()
+ mock_query_context.queries = [mock_query_object]
+ mock_query_context.query_cache_key.return_value = "referenced-chart-key"
+ mock_annotation_chart.get_query_context.return_value = mock_query_context
+ with patch(
+ "superset.common.query_context_processor.security_manager",
+ new_callable=MagicMock,
+ ) as security_manager:
+ security_manager.can_access_datasource.return_value = True
+ scope = processor._annotation_source_scope({"value": 1})
+ assert scope == {"access": True, "data_key": ["referenced-chart-key"]}
+ mock_query_context.query_cache_key.assert_called_once_with(mock_query_object)
+
+
+def test_annotation_source_scope_uses_live_fetch_authorization(
+ processor, mock_annotation_chart
+) -> None:
+ """When the referenced chart has a saved query context, ``access`` must
+ come from that context's own ``raise_for_access`` -- the same
+ authorization path the live fetch in ``get_viz_annotation_data`` uses --
+ not the coarser, context-free ``can_access_datasource``. A requester
+ denied by ``can_access_datasource`` but granted via a bypass that depends
+ on the chart's own saved form_data (e.g. a dashboard/viewer-promiscuous
+ bypass) must get a scope distinct from one truly denied, or the latter
+ could read the former's cached payload."""
+ from superset.exceptions import SupersetSecurityException
Review Comment:
<div>
<div id="suggestion">
<div id="issue"><b>Inline import unjustified</b></div>
<div id="fix">
BITO adaptive rule 12745 requires module-level imports unless a documented circular dependency exists. `SupersetSecurityException` is imported inside the test body, yet `superset.exceptions` is already imported at module level (line 35 of this file), so no circular dependency is possible β extend that import instead.
</div>
</div>
<small><i>Code Review Run #a2d0a6</i></small>
</div>
---
Should Bito avoid suggestions like this for future reviews? (<a href=https://alpha.bito.ai/home/ai-agents/review-rules>Manage Rules</a>)
- [ ] Yes, avoid them
##########
tests/unit_tests/common/test_query_context_processor.py:
##########
@@ -2630,6 +3041,94 @@ def test_mark_force_executed_noop_without_nonce(processor, mock_query_context):
cache_manager.data_cache.set.assert_not_called()
+# =============================================================================
+# Annotation-data cache decoupled from the dataframe cache
+# =============================================================================
+
+
+def test_get_annotation_data_cached_reads_from_cache(processor):
+ """A hit on the annotation-specific key skips recomputation entirely."""
+ with patch(
+ "superset.common.query_context_processor.cache_manager"
+ ) as cache_manager:
+ cache_manager.data_cache.get.return_value = {"annotation_data": {"a": 1}}
+ with patch.object(processor, "get_annotation_data") as mock_get:
+ result = processor._get_annotation_data_cached(
+ query_obj=MagicMock(),
+ cache_key="ak",
+ force_query=False,
+ force_cached=False,
+ timeout=60,
+ datasource_uid="ds",
+ )
+ assert result == {"a": 1}
+ mock_get.assert_not_called()
+
+
+def test_get_annotation_data_cached_computes_and_caches_on_miss(processor):
Review Comment:
<div>
<div id="suggestion">
<div id="issue"><b>Missing test docstring</b></div>
<div id="fix">
BITO.md adaptive rule 12148 requires every new test function to carry a docstring; the sibling tests in this same diff (`..._reads_from_cache`, `..._force_cached_raises_on_miss`) have one, but `test_get_annotation_data_cached_computes_and_caches_on_miss` does not. Adding a one-line docstring keeps test intent visible and consistent across the section.
</div>
</div>
<small><i>Code Review Run #a2d0a6</i></small>
</div>
---
Should Bito avoid suggestions like this for future reviews? (<a href=https://alpha.bito.ai/home/ai-agents/review-rules>Manage Rules</a>)
- [ ] Yes, avoid them
##########
tests/unit_tests/common/test_query_context_processor.py:
##########
@@ -117,25 +118,259 @@ def processor(mock_query_context):
return processor
-def test_query_cache_key_binds_annotation_data_to_requesting_user(processor):
- """The cache key for annotated queries must differ per requesting user."""
+def test_annotation_cache_key_binds_native_annotation_read_scope(processor) -> None:
+ """The annotation cache key for NATIVE layers must differ when the
+ requester's ``can_read`` (Annotation) access differs -- not who they are."""
query_obj = MagicMock()
query_obj.annotation_layers = [{"sourceType": "NATIVE", "name": "a", "value": 1}]
- with (
- patch(
- "superset.common.query_context_processor.get_user_id",
- side_effect=[1, 2],
- ),
- patch("superset.common.query_context_processor.security_manager"),
- ):
- processor.query_cache_key(query_obj)
- processor.query_cache_key(query_obj)
+ # ``security_manager`` autodetects as an async spec under a bare
+ # ``patch()`` (its real object trips ``unittest.mock``'s coroutine
+ # inference), which would silently turn every attribute access into an
+ # ``AsyncMock`` returning a fresh unawaited coroutine per call -- always
+ # unequal to itself and never equal to a configured return value. Forcing
+ # ``new_callable=MagicMock`` keeps these synchronous, as the real object
+ # is.
+ with patch(
+ "superset.common.query_context_processor.security_manager",
+ new_callable=MagicMock,
+ ) as security_manager:
+ security_manager.can_access.side_effect = [True, False]
+ processor.annotation_cache_key(query_obj)
+ processor.annotation_cache_key(query_obj)
contexts = [
call.kwargs["annotation_context"] for call in query_obj.cache_key.call_args_list
]
assert contexts[0] != contexts[1]
+def test_annotation_cache_key_shares_across_same_access_scope() -> None:
+ """Two distinct requesters (separate processor/query-object instances,
+ standing in for two different requests) with identical access scope must
+ produce identical annotation-context material. Reusing a single
+ processor/query_obj across both calls (as this test previously did)
+ would pass trivially regardless of whether the key is scope-based or
+ identity-based, since nothing about "who's asking" would ever vary."""
+ layer = {"sourceType": "NATIVE", "name": "a", "value": 1}
+ processor_a = QueryContextProcessor(MagicMock())
+ processor_b = QueryContextProcessor(MagicMock())
+ query_obj_a = MagicMock(annotation_layers=[layer])
+ query_obj_b = MagicMock(annotation_layers=[layer])
+
+ with patch(
+ "superset.common.query_context_processor.security_manager",
+ new_callable=MagicMock,
+ ) as security_manager:
+ security_manager.can_access.return_value = True
+ context_a = processor_a._annotation_cache_context(query_obj_a)
+ context_b = processor_b._annotation_cache_context(query_obj_b)
+
+ assert context_a == context_b
+
+
+def test_query_cache_key_does_not_bind_annotation_scope(processor) -> None:
+ """The dataframe cache key must stay shared across viewers of the same
+ chart, even when the query has annotation layers β only the separate
+ annotation cache key (see above) carries access-scope material."""
+ query_obj = MagicMock()
+ query_obj.annotation_layers = [{"sourceType": "NATIVE", "name": "a", "value": 1}]
+ with patch(
+ "superset.common.query_context_processor.security_manager",
+ new_callable=MagicMock,
+ ):
+ processor.query_cache_key(query_obj)
+ processor.query_cache_key(query_obj)
+ for call in query_obj.cache_key.call_args_list:
+ assert "annotation_context" not in call.kwargs
+
+
+@pytest.fixture
+def mock_annotation_chart() -> Iterator[MagicMock]:
+ """A found chart, wired as the referenced chart for
+ ``_annotation_source_scope`` tests -- factors out the repeated
+ ``ChartDAO.find_by_id`` patch those tests all need."""
+ chart = MagicMock()
+ with patch(
+ "superset.common.query_context_processor.ChartDAO.find_by_id",
+ return_value=chart,
+ ):
+ yield chart
+
+
+def test_annotation_source_scope_binds_datasource_access(
+ processor, mock_annotation_chart
+) -> None:
+ """A chart-backed annotation layer's scope must differ when the
+ requester's access to the referenced datasource differs."""
+ mock_annotation_chart.get_query_context.return_value = None
+ with patch(
+ "superset.common.query_context_processor.security_manager",
+ new_callable=MagicMock,
+ ) as security_manager:
+ security_manager.can_access_datasource.side_effect = [True, False]
+ security_manager.get_rls_cache_key.return_value = []
+ scope_a = processor._annotation_source_scope({"value": 1})
+ scope_b = processor._annotation_source_scope({"value": 1})
+ assert scope_a != scope_b
+ assert scope_a["access"] is True
+ assert scope_b["access"] is False
+
+
+def test_annotation_source_scope_reuses_referenced_chart_cache_key(
+ processor, mock_annotation_chart
+) -> None:
+ """When the referenced chart has a saved query context, its own cache
+ key(s) -- covering RLS and per-user Jinja/virtual-dataset material -- are
+ reused rather than re-derived."""
+ mock_query_object = MagicMock()
+ mock_query_context = MagicMock()
+ mock_query_context.queries = [mock_query_object]
+ mock_query_context.query_cache_key.return_value = "referenced-chart-key"
+ mock_annotation_chart.get_query_context.return_value = mock_query_context
+ with patch(
+ "superset.common.query_context_processor.security_manager",
+ new_callable=MagicMock,
+ ) as security_manager:
+ security_manager.can_access_datasource.return_value = True
+ scope = processor._annotation_source_scope({"value": 1})
+ assert scope == {"access": True, "data_key": ["referenced-chart-key"]}
+ mock_query_context.query_cache_key.assert_called_once_with(mock_query_object)
+
+
+def test_annotation_source_scope_uses_live_fetch_authorization(
+ processor, mock_annotation_chart
+) -> None:
+ """When the referenced chart has a saved query context, ``access`` must
+ come from that context's own ``raise_for_access`` -- the same
+ authorization path the live fetch in ``get_viz_annotation_data`` uses --
+ not the coarser, context-free ``can_access_datasource``. A requester
+ denied by ``can_access_datasource`` but granted via a bypass that depends
+ on the chart's own saved form_data (e.g. a dashboard/viewer-promiscuous
+ bypass) must get a scope distinct from one truly denied, or the latter
+ could read the former's cached payload."""
+ from superset.exceptions import SupersetSecurityException
+
+ mock_query_object = MagicMock()
+ mock_query_context = MagicMock()
Review Comment:
<div>
<div id="suggestion">
<div id="issue"><b>Mock vars unannotated</b></div>
<div id="fix">
Same BITO 12787 violation as lines 224-225: `mock_query_object` and `mock_query_context` are bare `MagicMock()` assignments in this new test. Annotating keeps the mock type visible to readers and static checkers.
</div>
</div>
<small><i>Code Review Run #a2d0a6</i></small>
</div>
---
Should Bito avoid suggestions like this for future reviews? (<a href=https://alpha.bito.ai/home/ai-agents/review-rules>Manage Rules</a>)
- [ ] Yes, avoid them
##########
tests/unit_tests/common/test_query_context_processor.py:
##########
@@ -117,25 +118,259 @@ def processor(mock_query_context):
return processor
-def test_query_cache_key_binds_annotation_data_to_requesting_user(processor):
- """The cache key for annotated queries must differ per requesting user."""
+def test_annotation_cache_key_binds_native_annotation_read_scope(processor) -> None:
+ """The annotation cache key for NATIVE layers must differ when the
+ requester's ``can_read`` (Annotation) access differs -- not who they are."""
query_obj = MagicMock()
query_obj.annotation_layers = [{"sourceType": "NATIVE", "name": "a", "value": 1}]
- with (
- patch(
- "superset.common.query_context_processor.get_user_id",
- side_effect=[1, 2],
- ),
- patch("superset.common.query_context_processor.security_manager"),
- ):
- processor.query_cache_key(query_obj)
- processor.query_cache_key(query_obj)
+ # ``security_manager`` autodetects as an async spec under a bare
+ # ``patch()`` (its real object trips ``unittest.mock``'s coroutine
+ # inference), which would silently turn every attribute access into an
+ # ``AsyncMock`` returning a fresh unawaited coroutine per call -- always
+ # unequal to itself and never equal to a configured return value. Forcing
+ # ``new_callable=MagicMock`` keeps these synchronous, as the real object
+ # is.
+ with patch(
+ "superset.common.query_context_processor.security_manager",
+ new_callable=MagicMock,
+ ) as security_manager:
+ security_manager.can_access.side_effect = [True, False]
+ processor.annotation_cache_key(query_obj)
+ processor.annotation_cache_key(query_obj)
contexts = [
call.kwargs["annotation_context"] for call in query_obj.cache_key.call_args_list
]
assert contexts[0] != contexts[1]
+def test_annotation_cache_key_shares_across_same_access_scope() -> None:
+ """Two distinct requesters (separate processor/query-object instances,
+ standing in for two different requests) with identical access scope must
+ produce identical annotation-context material. Reusing a single
+ processor/query_obj across both calls (as this test previously did)
+ would pass trivially regardless of whether the key is scope-based or
+ identity-based, since nothing about "who's asking" would ever vary."""
+ layer = {"sourceType": "NATIVE", "name": "a", "value": 1}
+ processor_a = QueryContextProcessor(MagicMock())
+ processor_b = QueryContextProcessor(MagicMock())
+ query_obj_a = MagicMock(annotation_layers=[layer])
+ query_obj_b = MagicMock(annotation_layers=[layer])
+
+ with patch(
+ "superset.common.query_context_processor.security_manager",
+ new_callable=MagicMock,
+ ) as security_manager:
+ security_manager.can_access.return_value = True
+ context_a = processor_a._annotation_cache_context(query_obj_a)
+ context_b = processor_b._annotation_cache_context(query_obj_b)
+
+ assert context_a == context_b
+
+
+def test_query_cache_key_does_not_bind_annotation_scope(processor) -> None:
+ """The dataframe cache key must stay shared across viewers of the same
+ chart, even when the query has annotation layers β only the separate
+ annotation cache key (see above) carries access-scope material."""
+ query_obj = MagicMock()
+ query_obj.annotation_layers = [{"sourceType": "NATIVE", "name": "a", "value": 1}]
+ with patch(
+ "superset.common.query_context_processor.security_manager",
+ new_callable=MagicMock,
+ ):
+ processor.query_cache_key(query_obj)
+ processor.query_cache_key(query_obj)
+ for call in query_obj.cache_key.call_args_list:
+ assert "annotation_context" not in call.kwargs
+
+
+@pytest.fixture
+def mock_annotation_chart() -> Iterator[MagicMock]:
+ """A found chart, wired as the referenced chart for
+ ``_annotation_source_scope`` tests -- factors out the repeated
+ ``ChartDAO.find_by_id`` patch those tests all need."""
+ chart = MagicMock()
+ with patch(
+ "superset.common.query_context_processor.ChartDAO.find_by_id",
+ return_value=chart,
+ ):
+ yield chart
+
+
+def test_annotation_source_scope_binds_datasource_access(
+ processor, mock_annotation_chart
+) -> None:
+ """A chart-backed annotation layer's scope must differ when the
+ requester's access to the referenced datasource differs."""
+ mock_annotation_chart.get_query_context.return_value = None
+ with patch(
+ "superset.common.query_context_processor.security_manager",
+ new_callable=MagicMock,
+ ) as security_manager:
+ security_manager.can_access_datasource.side_effect = [True, False]
+ security_manager.get_rls_cache_key.return_value = []
+ scope_a = processor._annotation_source_scope({"value": 1})
+ scope_b = processor._annotation_source_scope({"value": 1})
+ assert scope_a != scope_b
+ assert scope_a["access"] is True
+ assert scope_b["access"] is False
+
+
+def test_annotation_source_scope_reuses_referenced_chart_cache_key(
+ processor, mock_annotation_chart
+) -> None:
+ """When the referenced chart has a saved query context, its own cache
+ key(s) -- covering RLS and per-user Jinja/virtual-dataset material -- are
+ reused rather than re-derived."""
+ mock_query_object = MagicMock()
+ mock_query_context = MagicMock()
Review Comment:
<div>
<div id="suggestion">
<div id="issue"><b>Mock vars unannotated</b></div>
<div id="fix">
BITO adaptive rule 12787 requires explicit `: MagicMock` annotations on mock variables in test files. `mock_query_object` and `mock_query_context` are bare assignments here; the identical pattern repeats at lines 252-253 in the second test. Annotating keeps the mock type visible to readers and static checkers.
</div>
</div>
<small><i>Code Review Run #a2d0a6</i></small>
</div>
---
Should Bito avoid suggestions like this for future reviews? (<a href=https://alpha.bito.ai/home/ai-agents/review-rules>Manage Rules</a>)
- [ ] Yes, avoid them
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] fix(charts): decouple annotation cache from the shared dataframe cache [superset]
Posted by "rusackas (via GitHub)" <gi...@apache.org>.
rusackas commented on code in PR #44406:
URL: https://github.com/apache/superset/pull/44406#discussion_r4216846589
##########
superset/common/query_context_processor.py:
##########
@@ -449,34 +481,189 @@ def query_cache_key(self, query_obj: QueryObject, **kwargs: Any) -> str | None:
)
return cache_key
- def _annotation_cache_context(self, query_obj: QueryObject) -> dict[str, Any]:
+ def annotation_cache_key(self, query_obj: QueryObject) -> str | None:
+ """
+ Cache key for this query's annotation-layer payload, or ``None`` when
+ the query has no annotation layers.
+
+ Annotation payloads are fetched under the requesting user's access
+ scope, which is a stricter security requirement than the dataframe
+ itself has. Keying them separately from :meth:`query_cache_key` keeps
+ that scoping from forcing every distinct viewer of an annotated chart
+ onto their own full copy of the (potentially much larger) shared
+ dataframe: users with the same access scope share this key too.
"""
- Cache-key material binding cached annotation data to its security
- context.
+ if not query_obj or not query_obj.annotation_layers:
+ return None
+ return self.query_cache_key(
+ query_obj, annotation_context=self._annotation_cache_context(query_obj)
+ )
- Annotation payloads are fetched per requesting user and stored on the
- same cache entry as the dataframe, so the key also binds the requesting
- user and, for chart-backed layers, the RLS clauses of the referenced
- chart's datasource.
+ def _annotation_cache_context(self, query_obj: QueryObject) -> dict[str, Any]:
+ """
+ Cache-key material binding annotation data to its security *scope* so
+ users with the same access share a cache entry and users with a
+ different scope β or no access β never read each other's data.
+
+ * NATIVE layers: the ``can_read`` permission on ``Annotation``, the
+ only user-dependent dimension of these global records.
+ * Chart-backed (``line``/``table``) layers: see
+ :meth:`_annotation_source_scope`.
"""
- source_rls: dict[str, list[str] | None] = {}
+ context: dict[str, Any] = {}
+
+ if any(
+ layer.get("sourceType") == "NATIVE" for layer in query_obj.annotation_layers
+ ):
+ context["annotation_read"] = security_manager.can_access(
+ "can_read", "Annotation"
+ )
+
+ source_scope: dict[str, Any] = {}
for layer in query_obj.annotation_layers:
if (
layer.get("sourceType")
not in ANNOTATION_SOURCE_TYPES_WITH_CHART_REFERENCE
):
continue
layer_value = layer.get("value")
+ source_scope[str(layer_value)] = self._annotation_source_scope(layer)
Review Comment:
Good catch @sadpandajoe, the scope is now recorded per layer rather than per chart ID, so two layers on the same chart with different overrides each keep their own entry in the key. Added a test for exactly that pair.
##########
superset/common/query_context_processor.py:
##########
@@ -449,34 +481,189 @@ def query_cache_key(self, query_obj: QueryObject, **kwargs: Any) -> str | None:
)
return cache_key
- def _annotation_cache_context(self, query_obj: QueryObject) -> dict[str, Any]:
+ def annotation_cache_key(self, query_obj: QueryObject) -> str | None:
+ """
+ Cache key for this query's annotation-layer payload, or ``None`` when
+ the query has no annotation layers.
+
+ Annotation payloads are fetched under the requesting user's access
+ scope, which is a stricter security requirement than the dataframe
+ itself has. Keying them separately from :meth:`query_cache_key` keeps
+ that scoping from forcing every distinct viewer of an annotated chart
+ onto their own full copy of the (potentially much larger) shared
+ dataframe: users with the same access scope share this key too.
"""
- Cache-key material binding cached annotation data to its security
- context.
+ if not query_obj or not query_obj.annotation_layers:
+ return None
+ return self.query_cache_key(
+ query_obj, annotation_context=self._annotation_cache_context(query_obj)
+ )
- Annotation payloads are fetched per requesting user and stored on the
- same cache entry as the dataframe, so the key also binds the requesting
- user and, for chart-backed layers, the RLS clauses of the referenced
- chart's datasource.
+ def _annotation_cache_context(self, query_obj: QueryObject) -> dict[str, Any]:
+ """
+ Cache-key material binding annotation data to its security *scope* so
+ users with the same access share a cache entry and users with a
+ different scope β or no access β never read each other's data.
+
+ * NATIVE layers: the ``can_read`` permission on ``Annotation``, the
+ only user-dependent dimension of these global records.
+ * Chart-backed (``line``/``table``) layers: see
+ :meth:`_annotation_source_scope`.
"""
- source_rls: dict[str, list[str] | None] = {}
+ context: dict[str, Any] = {}
+
+ if any(
+ layer.get("sourceType") == "NATIVE" for layer in query_obj.annotation_layers
+ ):
+ context["annotation_read"] = security_manager.can_access(
+ "can_read", "Annotation"
+ )
+
+ source_scope: dict[str, Any] = {}
for layer in query_obj.annotation_layers:
if (
layer.get("sourceType")
not in ANNOTATION_SOURCE_TYPES_WITH_CHART_REFERENCE
):
continue
layer_value = layer.get("value")
+ source_scope[str(layer_value)] = self._annotation_source_scope(layer)
+ if source_scope:
+ context["source_scope"] = source_scope
+
+ return context
+
+ def _annotation_source_scope(self, layer: dict[str, Any]) -> dict[str, Any]:
+ """
+ Access and data-identity cache-key material for one chart-backed
+ annotation layer.
+
+ ``access`` keeps a user denied the referenced chart's datasource from
+ reading an authorized user's cached payload. When the chart has a
+ saved query context, this runs the *same* authorization path
+ :meth:`get_viz_annotation_data` executes
+ (``QueryContext.raise_for_access``) rather than the coarser
+ :meth:`SecurityManager.can_access_datasource` β a requester whose
+ access comes from a dashboard/viewer-promiscuous-mode bypass (which
+ depends on the chart's own saved ``form_data``, e.g. its
+ ``slice_id``/``dashboardId``) would otherwise still fail that coarser,
+ context-free check and collapse onto the same denied scope as a
+ genuinely unauthorized requester, letting the latter read the
+ former's cached payload. ``data_key`` is the annotation chart's own
+ query cache key(s) β derived from the same, override-applied query
+ objects actually executed (see :meth:`_apply_annotation_overrides`),
+ so it captures the datasource version, RLS clauses, and any per-user
+ Jinja/virtual-dataset RLS material exactly as the live fetch would,
+ including material an override only introduces at a finer grain.
+ Reusing this logic avoids re-deriving it and automatically inherits
+ any future correctness fixes made there.
+ """
+ layer_value = layer.get("value")
+ datasource = None
+ try:
chart = (
ChartDAO.find_by_id(layer_value) if layer_value is not None else None
)
- annotation_datasource = chart.datasource if chart else None
- source_rls[str(layer.get("value"))] = (
- security_manager.get_rls_cache_key(annotation_datasource)
- if annotation_datasource
- else None
+ # resolved_datasource, not datasource: the latter is pinned to
+ # table-backed datasources and resolves to None for a
+ # semantic-view-backed chart, which would otherwise collapse
+ # every requester onto the same {access: None, data_key: None}
+ # scope below regardless of their actual access.
+ datasource = chart.resolved_datasource if chart else None
+ if chart is None or datasource is None:
+ return {"access": None, "data_key": None}
Review Comment:
Agreed @sadpandajoe, an unresolved scope is now a no-cache signal rather than a placeholder key: the payload gets fetched live and never written. The cleared-datasource case also goes through the saved query context's own `raise_for_access` now, same as the live fetch, so allowed and denied viewers no longer land on the same key.
##########
superset/common/query_context_processor.py:
##########
@@ -449,34 +481,189 @@ def query_cache_key(self, query_obj: QueryObject, **kwargs: Any) -> str | None:
)
return cache_key
- def _annotation_cache_context(self, query_obj: QueryObject) -> dict[str, Any]:
+ def annotation_cache_key(self, query_obj: QueryObject) -> str | None:
+ """
+ Cache key for this query's annotation-layer payload, or ``None`` when
+ the query has no annotation layers.
+
+ Annotation payloads are fetched under the requesting user's access
+ scope, which is a stricter security requirement than the dataframe
+ itself has. Keying them separately from :meth:`query_cache_key` keeps
+ that scoping from forcing every distinct viewer of an annotated chart
+ onto their own full copy of the (potentially much larger) shared
+ dataframe: users with the same access scope share this key too.
"""
- Cache-key material binding cached annotation data to its security
- context.
+ if not query_obj or not query_obj.annotation_layers:
+ return None
+ return self.query_cache_key(
+ query_obj, annotation_context=self._annotation_cache_context(query_obj)
+ )
- Annotation payloads are fetched per requesting user and stored on the
- same cache entry as the dataframe, so the key also binds the requesting
- user and, for chart-backed layers, the RLS clauses of the referenced
- chart's datasource.
+ def _annotation_cache_context(self, query_obj: QueryObject) -> dict[str, Any]:
+ """
+ Cache-key material binding annotation data to its security *scope* so
+ users with the same access share a cache entry and users with a
+ different scope β or no access β never read each other's data.
+
+ * NATIVE layers: the ``can_read`` permission on ``Annotation``, the
+ only user-dependent dimension of these global records.
+ * Chart-backed (``line``/``table``) layers: see
+ :meth:`_annotation_source_scope`.
"""
- source_rls: dict[str, list[str] | None] = {}
+ context: dict[str, Any] = {}
+
+ if any(
+ layer.get("sourceType") == "NATIVE" for layer in query_obj.annotation_layers
+ ):
+ context["annotation_read"] = security_manager.can_access(
+ "can_read", "Annotation"
+ )
+
+ source_scope: dict[str, Any] = {}
for layer in query_obj.annotation_layers:
if (
layer.get("sourceType")
not in ANNOTATION_SOURCE_TYPES_WITH_CHART_REFERENCE
):
continue
layer_value = layer.get("value")
+ source_scope[str(layer_value)] = self._annotation_source_scope(layer)
+ if source_scope:
+ context["source_scope"] = source_scope
+
+ return context
+
+ def _annotation_source_scope(self, layer: dict[str, Any]) -> dict[str, Any]:
+ """
+ Access and data-identity cache-key material for one chart-backed
+ annotation layer.
+
+ ``access`` keeps a user denied the referenced chart's datasource from
+ reading an authorized user's cached payload. When the chart has a
+ saved query context, this runs the *same* authorization path
+ :meth:`get_viz_annotation_data` executes
+ (``QueryContext.raise_for_access``) rather than the coarser
+ :meth:`SecurityManager.can_access_datasource` β a requester whose
+ access comes from a dashboard/viewer-promiscuous-mode bypass (which
+ depends on the chart's own saved ``form_data``, e.g. its
+ ``slice_id``/``dashboardId``) would otherwise still fail that coarser,
+ context-free check and collapse onto the same denied scope as a
+ genuinely unauthorized requester, letting the latter read the
+ former's cached payload. ``data_key`` is the annotation chart's own
+ query cache key(s) β derived from the same, override-applied query
+ objects actually executed (see :meth:`_apply_annotation_overrides`),
+ so it captures the datasource version, RLS clauses, and any per-user
+ Jinja/virtual-dataset RLS material exactly as the live fetch would,
+ including material an override only introduces at a finer grain.
+ Reusing this logic avoids re-deriving it and automatically inherits
+ any future correctness fixes made there.
+ """
+ layer_value = layer.get("value")
+ datasource = None
+ try:
chart = (
ChartDAO.find_by_id(layer_value) if layer_value is not None else None
)
- annotation_datasource = chart.datasource if chart else None
- source_rls[str(layer.get("value"))] = (
- security_manager.get_rls_cache_key(annotation_datasource)
- if annotation_datasource
- else None
+ # resolved_datasource, not datasource: the latter is pinned to
+ # table-backed datasources and resolves to None for a
+ # semantic-view-backed chart, which would otherwise collapse
+ # every requester onto the same {access: None, data_key: None}
+ # scope below regardless of their actual access.
+ datasource = chart.resolved_datasource if chart else None
+ if chart is None or datasource is None:
+ return {"access": None, "data_key": None}
+
+ annotation_query_context = chart.get_query_context()
+ if annotation_query_context is not None:
+ self._apply_annotation_overrides(annotation_query_context, layer)
+ try:
+ annotation_query_context.raise_for_access()
+ access: Any = True
+ except SupersetSecurityException:
+ access = False
+ data_key: Any = [
Review Comment:
Good call @sadpandajoe. The override now sets `time_range` on the query objects too, so `cache_key` keys on the logical range and drops the resolved bounds, same as a saved range would. Added an assertion for it in the overrides test.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] fix(charts): decouple annotation cache from the shared dataframe cache [superset]
Posted by "sadpandajoe (via GitHub)" <gi...@apache.org>.
sadpandajoe commented on code in PR #44406:
URL: https://github.com/apache/superset/pull/44406#discussion_r4075812948
##########
superset/common/query_context_processor.py:
##########
@@ -328,6 +327,26 @@ def get_df_payload_result(
# nonce reads the freshly-cached result instead of recomputing it.
self._mark_force_executed(query_obj, cache_key, cache.result_persisted)
+ # Annotation data is fetched per requesting user (and, for chart-backed
+ # layers, scoped by the referenced chart datasource's RLS), so it is
+ # resolved and cached under its own entry β independent of whether the
+ # (shareable) dataframe above was a hit or a miss β rather than forcing
+ # every viewer of the same chart onto their own full dataframe copy.
+ annotation_data: dict[str, Any] = {}
+ if query_obj and annotation_key and cache.status != QueryStatus.FAILED:
+ try:
+ annotation_data = self._get_annotation_data_cached(
+ query_obj=query_obj,
+ cache_key=annotation_key,
+ force_query=force_query,
Review Comment:
A forced refresh can still serve stale annotations for a second access scope: once the shared dataframe marker suppresses `force_query`, this reads that scope's existing annotation entry instead of refreshing it. Should the annotation cache have its own force/nonce state so each scoped entry is refreshed?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] fix(charts): decouple annotation cache from the shared dataframe cache [superset]
Posted by "rusackas (via GitHub)" <gi...@apache.org>.
rusackas commented on PR #44406:
URL: https://github.com/apache/superset/pull/44406#issuecomment-5802049328
Pushed two small commits from review on #44546, which had been stacked on this branch by mistake: the chart/datasource lookup now sits inside the fail-closed `try`, plus a caplog assertion on the warning.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] fix(charts): decouple annotation cache from the shared dataframe cache [superset]
Posted by "sadpandajoe (via GitHub)" <gi...@apache.org>.
sadpandajoe commented on code in PR #44406:
URL: https://github.com/apache/superset/pull/44406#discussion_r4290176618
##########
superset/common/query_context_processor.py:
##########
@@ -354,6 +359,30 @@ def get_df_payload_result(
# nonce reads the freshly-cached result instead of recomputing it.
self._mark_force_executed(query_obj, cache_key, cache.result_persisted)
+ # Annotation data is fetched per requesting user (and, for chart-backed
+ # layers, scoped by the referenced chart datasource's RLS), so it is
+ # resolved and cached under its own entry β independent of whether the
+ # (shareable) dataframe above was a hit or a miss β rather than forcing
+ # every viewer of the same chart onto their own full dataframe copy.
+ annotation_data: dict[str, Any] = {}
+ if (
+ query_obj
+ and query_obj.annotation_layers
+ and cache.status != QueryStatus.FAILED
+ ):
+ try:
+ annotation_data = self._get_annotation_data_cached(
+ query_obj=query_obj,
+ cache_key=annotation_key,
+ force_query=annotation_force_query,
+ force_cached=force_cached,
+ timeout=self.get_cache_timeout(),
+ datasource_uid=self._qc_datasource.uid,
+ )
+ except QueryObjectValidationError as ex:
Review Comment:
Annotation fetches can now lose their incomplete-result signal: `SemanticResultCompletenessError` subclasses `QueryObjectValidationError`, so when the dataframe succeeds but a semantic-view annotation raises it here, this catch turns it into a failed payload instead of letting it propagate as the dataframe path above does (and as the old combined block did). `execute_chart_query` then returns normally and the async task is marked successful without publishing `semantic_result_error`. Should this re-raise `SemanticResultCompletenessError` before the generic catch, with a `get_df_payload_result` test asserting an annotation-side completeness error escapes?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] fix(charts): decouple annotation cache from the shared dataframe cache [superset]
Posted by "sadpandajoe (via GitHub)" <gi...@apache.org>.
sadpandajoe commented on code in PR #44406:
URL: https://github.com/apache/superset/pull/44406#discussion_r4320138868
##########
superset/charts/data/api.py:
##########
@@ -448,6 +448,15 @@ def _run_async(
return self._send_chart_response(result)
except ChartDataCacheLoadError:
pass
+ except (
Review Comment:
This new 400 branch for a failed cached read-back has no test. A warm dataframe whose annotation layer references a chart the viewer cannot access reaches it, but the existing `_run_async` tests only cover a cache hit and a cache miss, so deleting this handler would leave them green and the request would return a 500. Could a test make `command.execute(force_cached=True)` raise `ChartDataQueryFailedError` and assert a 400 with no task submitted?
##########
superset/common/query_context_processor.py:
##########
@@ -331,6 +342,26 @@ def get_df_payload_result(
# nonce reads the freshly-cached result instead of recomputing it.
self._mark_force_executed(query_obj, cache_key, cache.result_persisted)
+ # Annotation data is fetched per requesting user (and, for chart-backed
+ # layers, scoped by the referenced chart datasource's RLS), so it is
+ # resolved and cached under its own entry β independent of whether the
+ # (shareable) dataframe above was a hit or a miss β rather than forcing
+ # every viewer of the same chart onto their own full dataframe copy.
+ annotation_data: dict[str, Any] = {}
+ if query_obj and annotation_key and cache.status != QueryStatus.FAILED:
+ try:
+ annotation_data = self._get_annotation_data_cached(
Review Comment:
The read-back after a joined task succeeds does not go through the `force_cached` probe: `requestChartDataResolved` reissues it with `enableAsyncMode: false`, so `_run_async` is never entered and viewer B's missing annotation entry is computed inline with `force_cached=False`. A slow annotation query can therefore still exceed the request timeout after the task reported success. Should the annotation scope be part of the task identity, or be warmed by the task, so each viewer's entry exists before completion?
##########
superset/common/query_context_processor.py:
##########
@@ -449,34 +481,189 @@ def query_cache_key(self, query_obj: QueryObject, **kwargs: Any) -> str | None:
)
return cache_key
- def _annotation_cache_context(self, query_obj: QueryObject) -> dict[str, Any]:
+ def annotation_cache_key(self, query_obj: QueryObject) -> str | None:
+ """
+ Cache key for this query's annotation-layer payload, or ``None`` when
+ the query has no annotation layers.
+
+ Annotation payloads are fetched under the requesting user's access
+ scope, which is a stricter security requirement than the dataframe
+ itself has. Keying them separately from :meth:`query_cache_key` keeps
+ that scoping from forcing every distinct viewer of an annotated chart
+ onto their own full copy of the (potentially much larger) shared
+ dataframe: users with the same access scope share this key too.
"""
- Cache-key material binding cached annotation data to its security
- context.
+ if not query_obj or not query_obj.annotation_layers:
+ return None
+ return self.query_cache_key(
+ query_obj, annotation_context=self._annotation_cache_context(query_obj)
+ )
- Annotation payloads are fetched per requesting user and stored on the
- same cache entry as the dataframe, so the key also binds the requesting
- user and, for chart-backed layers, the RLS clauses of the referenced
- chart's datasource.
+ def _annotation_cache_context(self, query_obj: QueryObject) -> dict[str, Any]:
+ """
+ Cache-key material binding annotation data to its security *scope* so
+ users with the same access share a cache entry and users with a
+ different scope β or no access β never read each other's data.
+
+ * NATIVE layers: the ``can_read`` permission on ``Annotation``, the
+ only user-dependent dimension of these global records.
+ * Chart-backed (``line``/``table``) layers: see
+ :meth:`_annotation_source_scope`.
"""
- source_rls: dict[str, list[str] | None] = {}
+ context: dict[str, Any] = {}
+
+ if any(
+ layer.get("sourceType") == "NATIVE" for layer in query_obj.annotation_layers
+ ):
+ context["annotation_read"] = security_manager.can_access(
+ "can_read", "Annotation"
+ )
+
+ source_scope: dict[str, Any] = {}
for layer in query_obj.annotation_layers:
if (
layer.get("sourceType")
not in ANNOTATION_SOURCE_TYPES_WITH_CHART_REFERENCE
):
continue
layer_value = layer.get("value")
+ source_scope[str(layer_value)] = self._annotation_source_scope(layer)
+ if source_scope:
+ context["source_scope"] = source_scope
+
+ return context
+
+ def _annotation_source_scope(self, layer: dict[str, Any]) -> dict[str, Any]:
+ """
+ Access and data-identity cache-key material for one chart-backed
+ annotation layer.
+
+ ``access`` keeps a user denied the referenced chart's datasource from
+ reading an authorized user's cached payload. When the chart has a
+ saved query context, this runs the *same* authorization path
+ :meth:`get_viz_annotation_data` executes
+ (``QueryContext.raise_for_access``) rather than the coarser
+ :meth:`SecurityManager.can_access_datasource` β a requester whose
+ access comes from a dashboard/viewer-promiscuous-mode bypass (which
+ depends on the chart's own saved ``form_data``, e.g. its
+ ``slice_id``/``dashboardId``) would otherwise still fail that coarser,
+ context-free check and collapse onto the same denied scope as a
+ genuinely unauthorized requester, letting the latter read the
+ former's cached payload. ``data_key`` is the annotation chart's own
+ query cache key(s) β derived from the same, override-applied query
+ objects actually executed (see :meth:`_apply_annotation_overrides`),
+ so it captures the datasource version, RLS clauses, and any per-user
+ Jinja/virtual-dataset RLS material exactly as the live fetch would,
+ including material an override only introduces at a finer grain.
+ Reusing this logic avoids re-deriving it and automatically inherits
+ any future correctness fixes made there.
+ """
+ layer_value = layer.get("value")
+ datasource = None
+ try:
chart = (
ChartDAO.find_by_id(layer_value) if layer_value is not None else None
)
- annotation_datasource = chart.datasource if chart else None
- source_rls[str(layer.get("value"))] = (
- security_manager.get_rls_cache_key(annotation_datasource)
- if annotation_datasource
- else None
+ # resolved_datasource, not datasource: the latter is pinned to
+ # table-backed datasources and resolves to None for a
+ # semantic-view-backed chart, which would otherwise collapse
+ # every requester onto the same {access: None, data_key: None}
+ # scope below regardless of their actual access.
+ datasource = chart.resolved_datasource if chart else None
+ if chart is None or datasource is None:
+ return {"access": None, "data_key": None}
+
+ annotation_query_context = chart.get_query_context()
+ if annotation_query_context is not None:
+ self._apply_annotation_overrides(annotation_query_context, layer)
+ try:
+ annotation_query_context.raise_for_access()
+ access: Any = True
+ except SupersetSecurityException:
+ access = False
+ data_key: Any = [
+ annotation_query_context.query_cache_key(query_object)
Review Comment:
The concern is the referenced chart, not the host. When the source chart's saved query context has `result_type="samples"` and metrics, `data_key` is derived from the unprepared query, but the live fetch runs it through `_prepare_samples_query`, which clears the metrics. An RLS clause that calls `current_user_id()` only when metrics are absent gives both viewers the same key but user-specific rows, so the second viewer is served the first viewer's rows. Should the key be derived from the query after result-type preparation?
##########
superset/common/query_context_processor.py:
##########
@@ -474,73 +503,226 @@ def query_cache_key(self, query_obj: QueryObject, **kwargs: Any) -> str | None:
)
return cache_key
- def _annotation_cache_context(self, query_obj: QueryObject) -> dict[str, Any]:
+ def annotation_cache_key(self, query_obj: QueryObject) -> str | None:
"""
- Cache-key material binding cached annotation data to its security
- context.
-
- Annotation payloads are fetched per requesting user and stored on the
- same cache entry as the dataframe, so the key also binds the requesting
- user and, for chart-backed layers, the RLS clauses of the referenced
- chart's datasource.
+ Cache key for this query's annotation-layer payload, or ``None`` when
+ the query has no annotation layers or when some layer's access scope
+ could not be resolved (the payload is then never cached, see
+ :meth:`_annotation_source_scope`).
+
+ Annotation payloads are fetched under the requesting user's access
+ scope, which is a stricter security requirement than the dataframe
+ itself has. Keying them separately from :meth:`query_cache_key` keeps
+ that scoping from forcing every distinct viewer of an annotated chart
+ onto their own full copy of the (potentially much larger) shared
+ dataframe: users with the same access scope share this key too.
"""
- source_rls: dict[str, list[str] | None] = {}
- source_versions: dict[str, tuple[str, str]] = {}
- metadata_versions: dict[str, str] = {}
+ if not query_obj or not query_obj.annotation_layers:
+ return None
+ if (annotation_context := self._annotation_cache_context(query_obj)) is None:
+ return None
+ return self.query_cache_key(query_obj, annotation_context=annotation_context)
+
+ def _annotation_cache_context(
+ self, query_obj: QueryObject
+ ) -> dict[str, Any] | None:
+ """
+ Cache-key material binding annotation data to its security *scope* so
+ users with the same access share a cache entry and users with a
+ different scope β or no access β never read each other's data.
+
+ * NATIVE layers: the ``can_read`` permission on ``Annotation``, the
+ only user-dependent dimension of these global records.
+ * Chart-backed (``line``/``table``) layers: see
+ :meth:`_annotation_source_scope`, recorded per *layer* (not per
+ referenced chart) since two layers can reference the same chart
+ with different overrides, and so different scopes.
+
+ Returns ``None`` when any layer's scope cannot be resolved, meaning
+ the payload must not be cached at all.
+ """
+ context: dict[str, Any] = {}
+
+ if any(
+ layer.get("sourceType") == "NATIVE" for layer in query_obj.annotation_layers
+ ):
+ context["annotation_read"] = security_manager.can_access(
+ "can_read", "Annotation"
+ )
+
+ source_scope: list[dict[str, Any]] = []
for layer in query_obj.annotation_layers:
if (
layer.get("sourceType")
not in ANNOTATION_SOURCE_TYPES_WITH_CHART_REFERENCE
):
continue
- layer_value = layer.get("value")
+ if (scope := self._annotation_source_scope(layer)) is None:
+ return None
+ source_scope.append({"value": str(layer.get("value")), **scope})
+ if source_scope:
+ context["source_scope"] = source_scope
+
+ return context
+
+ def _annotation_source_scope(self, layer: dict[str, Any]) -> dict[str, Any] | None:
+ """
+ Access and data-identity cache-key material for one chart-backed
+ annotation layer, or ``None`` when it cannot be resolved.
+
+ ``None`` means "do not cache": a missing chart, a chart with neither
+ a saved query context nor a resolvable datasource, or any error while
+ deriving the scope would otherwise collapse an authorized and a
+ denied requester onto one shared identity, letting the latter read
+ the former's warmed payload. The caller fetches live instead.
+
+ ``access`` keeps a user denied the referenced chart's datasource from
+ reading an authorized user's cached payload. When the chart has a
+ saved query context, this runs the *same* authorization path
+ :meth:`get_viz_annotation_data` executes
+ (``QueryContext.raise_for_access``) rather than the coarser
+ :meth:`SecurityManager.can_access_datasource` β a requester whose
+ access comes from a dashboard/viewer-promiscuous-mode bypass (which
+ depends on the chart's own saved ``form_data``, e.g. its
+ ``slice_id``/``dashboardId``) would otherwise still fail that coarser,
+ context-free check and collapse onto the same denied scope as a
+ genuinely unauthorized requester, letting the latter read the
+ former's cached payload. ``data_key`` is the annotation chart's own
+ query cache key(s) β derived from the same, override-applied query
+ objects actually executed (see :meth:`_apply_annotation_overrides`),
+ so it captures the datasource version, RLS clauses, and any per-user
+ Jinja/virtual-dataset RLS material exactly as the live fetch would,
+ including material an override only introduces at a finer grain.
+ Reusing this logic avoids re-deriving it and automatically inherits
+ any future correctness fixes made there.
+ """
+ layer_value = layer.get("value")
+ try:
chart = (
ChartDAO.find_by_id(layer_value) if layer_value is not None else None
)
- annotation_datasource: Datasource | None = (
- chart.resolved_datasource if chart else None
- )
- if chart and isinstance(chart.query_context, str):
+ if chart is None:
+ return None
+
+ # The saved query context is what the live fetch executes and
+ # authorizes against, so it is consulted first, independent of
+ # the chart's own datasource pointer: a chart can have its
+ # ``datasource_id`` cleared while its saved query context still
+ # targets a valid dataset, and the live fetch would still work.
+ annotation_query_context = chart.get_query_context()
+ if annotation_query_context is not None:
+ self._apply_annotation_overrides(annotation_query_context, layer)
try:
- source: dict[str, Any] = json.loads(chart.query_context)[
- "datasource"
- ]
- annotation_datasource = DatasourceDAO.get_datasource(
- source["type"], source["id"]
- )
- except (
- ValueError,
- TypeError,
- KeyError,
- DatasourceNotFound,
- DatasourceTypeNotSupportedError,
- DatasourceValueIsIncorrect,
- ):
- annotation_datasource = None
+ annotation_query_context.raise_for_access()
+ access: Any = True
+ except SupersetSecurityException:
+ access = False
+ data_key: Any = [
+ annotation_query_context.query_cache_key(query_object)
Review Comment:
The comparison to the source chart's own cache doesn't hold when that chart sets `cache_timeout=-1`: it never caches, so nothing is shared there, but the annotation entry is still written under the host chart's TTL with the same key for both viewers. That serves the first viewer's total to the second even though the source opted out of caching. Should annotation caching be skipped when the referenced chart's cache is disabled?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] fix(charts): decouple annotation cache from the shared dataframe cache [superset]
Posted by "rusackas (via GitHub)" <gi...@apache.org>.
rusackas commented on code in PR #44406:
URL: https://github.com/apache/superset/pull/44406#discussion_r4045402367
##########
superset/common/query_context_processor.py:
##########
@@ -445,31 +464,144 @@ def query_cache_key(self, query_obj: QueryObject, **kwargs: Any) -> str | None:
)
return cache_key
- def _annotation_cache_context(self, query_obj: QueryObject) -> dict[str, Any]:
+ def annotation_cache_key(self, query_obj: QueryObject) -> str | None:
+ """
+ Cache key for this query's annotation-layer payload, or ``None`` when
+ the query has no annotation layers.
+
+ Annotation payloads are fetched under the requesting user's access
+ scope, which is a stricter security requirement than the dataframe
+ itself has. Keying them separately from :meth:`query_cache_key` keeps
+ that scoping from forcing every distinct viewer of an annotated chart
+ onto their own full copy of the (potentially much larger) shared
+ dataframe: users with the same access scope share this key too.
"""
- Cache-key material binding cached annotation data to its security
- context.
+ if not query_obj or not query_obj.annotation_layers:
+ return None
+ return self.query_cache_key(
+ query_obj, annotation_context=self._annotation_cache_context(query_obj)
+ )
- Annotation payloads are fetched per requesting user and stored on the
- same cache entry as the dataframe, so the key also binds the requesting
- user and, for chart-backed layers, the RLS clauses of the referenced
- chart's datasource.
+ def _annotation_cache_context(self, query_obj: QueryObject) -> dict[str, Any]:
+ """
+ Cache-key material binding annotation data to its security *scope* so
+ users with the same access share a cache entry and users with a
+ different scope β or no access β never read each other's data.
+
+ * NATIVE layers: the ``can_read`` permission on ``Annotation``, the
+ only user-dependent dimension of these global records.
+ * Chart-backed (``line``/``table``) layers: see
+ :meth:`_annotation_source_scope`.
"""
- source_rls: dict[str, list[str] | None] = {}
+ context: dict[str, Any] = {}
+
+ if any(
+ layer.get("sourceType") == "NATIVE" for layer in query_obj.annotation_layers
+ ):
+ context["annotation_read"] = security_manager.can_access(
+ "can_read", "Annotation"
+ )
+
+ source_scope: dict[str, Any] = {}
for layer in query_obj.annotation_layers:
if layer.get("sourceType") not in ("line", "table"):
continue
layer_value = layer.get("value")
- chart = (
- ChartDAO.find_by_id(layer_value) if layer_value is not None else None
+ source_scope[str(layer_value)] = self._annotation_source_scope(layer_value)
+ if source_scope:
+ context["source_scope"] = source_scope
+
+ return context
+
+ def _annotation_source_scope(self, layer_value: Any) -> dict[str, Any]:
+ """
+ Access and data-identity cache-key material for one chart-backed
+ annotation layer.
+
+ ``access`` keeps a user denied the referenced chart's datasource from
+ reading an authorized user's cached payload. ``data_key`` is the
+ annotation chart's own query cache key(s), which already capture the
+ datasource version, RLS clauses, and any per-user Jinja/virtual-dataset
+ RLS material β reusing it here avoids re-deriving that logic and
+ automatically inherits any future correctness fixes made there.
+ """
+ chart = ChartDAO.find_by_id(layer_value) if layer_value is not None else None
+ datasource = chart.datasource if chart else None
+ if chart is None or datasource is None:
+ return {"access": None, "data_key": None}
+
+ try:
+ access = security_manager.can_access_datasource(datasource)
+ # Fall back to the RLS-clause identity when the chart has no saved
+ # query context to key on.
+ annotation_query_context = chart.get_query_context()
+ data_key: Any = (
+ [
+ annotation_query_context.query_cache_key(query_object)
+ for query_object in annotation_query_context.queries
+ ]
+ if annotation_query_context is not None
+ else security_manager.get_rls_cache_key(datasource)
+ )
+ except SupersetException:
Review Comment:
Good catch, widened it to a bare except. Also found the fallback re-calls get_rls_cache_key, so if that's what failed in the first place it'd re-raise right out of the except block too. Wrapped that as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] fix(charts): decouple annotation cache from the shared dataframe cache [superset]
Posted by "bito-code-review[bot] (via GitHub)" <gi...@apache.org>.
bito-code-review[bot] commented on PR #44406:
URL: https://github.com/apache/superset/pull/44406#issuecomment-5804415982
AI Code Review is in progress (usually takes 3 to 15 minutes unless it's a very large PR).
<details>
<summary>
Bito Usage Guide
</summary>
**Commands**
Type the following command in the pull request comment and save the comment.
- `/review` - Manually triggers an incremental AI Review.
- `/review full` - Manually triggers a full AI Review.
- `/pause` - Pauses automatic reviews on this pull request.
- `/resume` - Resumes automatic reviews.
- `/resolve` - Marks all Bito-posted review comments as resolved.
- `/abort` - Cancels all in-progress reviews.
Refer to the <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/available-commands" target="_blank" rel="noopener noreferrer">documentation</a> for additional commands.
**Configuration**
This repository uses `Superset` You can customize the agent settings <a href="https://alpha.bito.ai/home/ai-agents/code-review-agent" target="_blank" rel="noopener noreferrer">here</a> or contact your Bito workspace admin at evan@preset.io.
**Documentation & Help**
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/getting-started/install-run-using-bito-cloud/create-or-customize-an-agent-instance" target="_blank" rel="noopener noreferrer">Customize agent settings</a>
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/implementing-custom-code-review-rules" target="_blank" rel="noopener noreferrer">Review rules</a>
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent" target="_blank" rel="noopener noreferrer">General documentation</a>
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/faqs" target="_blank" rel="noopener noreferrer">FAQ</a>
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] fix(charts): decouple annotation cache from the shared dataframe cache [superset]
Posted by "sadpandajoe (via GitHub)" <gi...@apache.org>.
sadpandajoe commented on code in PR #44406:
URL: https://github.com/apache/superset/pull/44406#discussion_r4172726408
##########
superset/common/query_context_processor.py:
##########
@@ -332,6 +344,26 @@ def get_df_payload_result(
# nonce reads the freshly-cached result instead of recomputing it.
self._mark_force_executed(query_obj, cache_key, cache.result_persisted)
+ # Annotation data is fetched per requesting user (and, for chart-backed
+ # layers, scoped by the referenced chart datasource's RLS), so it is
+ # resolved and cached under its own entry β independent of whether the
+ # (shareable) dataframe above was a hit or a miss β rather than forcing
+ # every viewer of the same chart onto their own full dataframe copy.
+ annotation_data: dict[str, Any] = {}
+ if query_obj and annotation_key and cache.status != QueryStatus.FAILED:
+ try:
+ annotation_data = self._get_annotation_data_cached(
Review Comment:
When the dataframe is cached but an annotation on another PostgreSQL server is not, the async task records that annotation cursor's PID against the host database, so abort/timeout targets the wrong server and can terminate an unrelated session. Should cancellation track the database of the query actually executing here?
##########
superset/common/query_context_processor.py:
##########
@@ -449,34 +481,189 @@ def query_cache_key(self, query_obj: QueryObject, **kwargs: Any) -> str | None:
)
return cache_key
- def _annotation_cache_context(self, query_obj: QueryObject) -> dict[str, Any]:
+ def annotation_cache_key(self, query_obj: QueryObject) -> str | None:
+ """
+ Cache key for this query's annotation-layer payload, or ``None`` when
+ the query has no annotation layers.
+
+ Annotation payloads are fetched under the requesting user's access
+ scope, which is a stricter security requirement than the dataframe
+ itself has. Keying them separately from :meth:`query_cache_key` keeps
+ that scoping from forcing every distinct viewer of an annotated chart
+ onto their own full copy of the (potentially much larger) shared
+ dataframe: users with the same access scope share this key too.
"""
- Cache-key material binding cached annotation data to its security
- context.
+ if not query_obj or not query_obj.annotation_layers:
+ return None
+ return self.query_cache_key(
+ query_obj, annotation_context=self._annotation_cache_context(query_obj)
+ )
- Annotation payloads are fetched per requesting user and stored on the
- same cache entry as the dataframe, so the key also binds the requesting
- user and, for chart-backed layers, the RLS clauses of the referenced
- chart's datasource.
+ def _annotation_cache_context(self, query_obj: QueryObject) -> dict[str, Any]:
+ """
+ Cache-key material binding annotation data to its security *scope* so
+ users with the same access share a cache entry and users with a
+ different scope β or no access β never read each other's data.
+
+ * NATIVE layers: the ``can_read`` permission on ``Annotation``, the
+ only user-dependent dimension of these global records.
+ * Chart-backed (``line``/``table``) layers: see
+ :meth:`_annotation_source_scope`.
"""
- source_rls: dict[str, list[str] | None] = {}
+ context: dict[str, Any] = {}
+
+ if any(
+ layer.get("sourceType") == "NATIVE" for layer in query_obj.annotation_layers
+ ):
+ context["annotation_read"] = security_manager.can_access(
+ "can_read", "Annotation"
+ )
+
+ source_scope: dict[str, Any] = {}
for layer in query_obj.annotation_layers:
if (
layer.get("sourceType")
not in ANNOTATION_SOURCE_TYPES_WITH_CHART_REFERENCE
):
continue
layer_value = layer.get("value")
+ source_scope[str(layer_value)] = self._annotation_source_scope(layer)
+ if source_scope:
+ context["source_scope"] = source_scope
+
+ return context
+
+ def _annotation_source_scope(self, layer: dict[str, Any]) -> dict[str, Any]:
+ """
+ Access and data-identity cache-key material for one chart-backed
+ annotation layer.
+
+ ``access`` keeps a user denied the referenced chart's datasource from
+ reading an authorized user's cached payload. When the chart has a
+ saved query context, this runs the *same* authorization path
+ :meth:`get_viz_annotation_data` executes
+ (``QueryContext.raise_for_access``) rather than the coarser
+ :meth:`SecurityManager.can_access_datasource` β a requester whose
+ access comes from a dashboard/viewer-promiscuous-mode bypass (which
+ depends on the chart's own saved ``form_data``, e.g. its
+ ``slice_id``/``dashboardId``) would otherwise still fail that coarser,
+ context-free check and collapse onto the same denied scope as a
+ genuinely unauthorized requester, letting the latter read the
+ former's cached payload. ``data_key`` is the annotation chart's own
+ query cache key(s) β derived from the same, override-applied query
+ objects actually executed (see :meth:`_apply_annotation_overrides`),
+ so it captures the datasource version, RLS clauses, and any per-user
+ Jinja/virtual-dataset RLS material exactly as the live fetch would,
+ including material an override only introduces at a finer grain.
+ Reusing this logic avoids re-deriving it and automatically inherits
+ any future correctness fixes made there.
+ """
+ layer_value = layer.get("value")
+ datasource = None
+ try:
chart = (
ChartDAO.find_by_id(layer_value) if layer_value is not None else None
)
- annotation_datasource = chart.datasource if chart else None
- source_rls[str(layer.get("value"))] = (
- security_manager.get_rls_cache_key(annotation_datasource)
- if annotation_datasource
- else None
+ # resolved_datasource, not datasource: the latter is pinned to
+ # table-backed datasources and resolves to None for a
+ # semantic-view-backed chart, which would otherwise collapse
+ # every requester onto the same {access: None, data_key: None}
+ # scope below regardless of their actual access.
+ datasource = chart.resolved_datasource if chart else None
+ if chart is None or datasource is None:
+ return {"access": None, "data_key": None}
+
+ annotation_query_context = chart.get_query_context()
+ if annotation_query_context is not None:
+ self._apply_annotation_overrides(annotation_query_context, layer)
+ try:
+ annotation_query_context.raise_for_access()
+ access: Any = True
+ except SupersetSecurityException:
+ access = False
+ data_key: Any = [
+ annotation_query_context.query_cache_key(query_object)
Review Comment:
A valid RLS clause such as `owner_id = {{ current_user_id () }}` is missed by the referenced query key's macro detector, so two users with the same grants get the same annotation key and the second receives the first user's rowsβeven if the referenced chart disables caching. Should annotation reuse require a verified user-dependent identity rather than trusting this syntactic detector?
##########
superset/common/query_context_processor.py:
##########
@@ -449,34 +481,189 @@ def query_cache_key(self, query_obj: QueryObject, **kwargs: Any) -> str | None:
)
return cache_key
- def _annotation_cache_context(self, query_obj: QueryObject) -> dict[str, Any]:
+ def annotation_cache_key(self, query_obj: QueryObject) -> str | None:
+ """
+ Cache key for this query's annotation-layer payload, or ``None`` when
+ the query has no annotation layers.
+
+ Annotation payloads are fetched under the requesting user's access
+ scope, which is a stricter security requirement than the dataframe
+ itself has. Keying them separately from :meth:`query_cache_key` keeps
+ that scoping from forcing every distinct viewer of an annotated chart
+ onto their own full copy of the (potentially much larger) shared
+ dataframe: users with the same access scope share this key too.
"""
- Cache-key material binding cached annotation data to its security
- context.
+ if not query_obj or not query_obj.annotation_layers:
+ return None
+ return self.query_cache_key(
+ query_obj, annotation_context=self._annotation_cache_context(query_obj)
+ )
- Annotation payloads are fetched per requesting user and stored on the
- same cache entry as the dataframe, so the key also binds the requesting
- user and, for chart-backed layers, the RLS clauses of the referenced
- chart's datasource.
+ def _annotation_cache_context(self, query_obj: QueryObject) -> dict[str, Any]:
+ """
+ Cache-key material binding annotation data to its security *scope* so
+ users with the same access share a cache entry and users with a
+ different scope β or no access β never read each other's data.
+
+ * NATIVE layers: the ``can_read`` permission on ``Annotation``, the
+ only user-dependent dimension of these global records.
+ * Chart-backed (``line``/``table``) layers: see
+ :meth:`_annotation_source_scope`.
"""
- source_rls: dict[str, list[str] | None] = {}
+ context: dict[str, Any] = {}
+
+ if any(
+ layer.get("sourceType") == "NATIVE" for layer in query_obj.annotation_layers
+ ):
+ context["annotation_read"] = security_manager.can_access(
+ "can_read", "Annotation"
+ )
+
+ source_scope: dict[str, Any] = {}
for layer in query_obj.annotation_layers:
if (
layer.get("sourceType")
not in ANNOTATION_SOURCE_TYPES_WITH_CHART_REFERENCE
):
continue
layer_value = layer.get("value")
+ source_scope[str(layer_value)] = self._annotation_source_scope(layer)
+ if source_scope:
+ context["source_scope"] = source_scope
+
+ return context
+
+ def _annotation_source_scope(self, layer: dict[str, Any]) -> dict[str, Any]:
+ """
+ Access and data-identity cache-key material for one chart-backed
+ annotation layer.
+
+ ``access`` keeps a user denied the referenced chart's datasource from
+ reading an authorized user's cached payload. When the chart has a
+ saved query context, this runs the *same* authorization path
+ :meth:`get_viz_annotation_data` executes
+ (``QueryContext.raise_for_access``) rather than the coarser
+ :meth:`SecurityManager.can_access_datasource` β a requester whose
+ access comes from a dashboard/viewer-promiscuous-mode bypass (which
+ depends on the chart's own saved ``form_data``, e.g. its
+ ``slice_id``/``dashboardId``) would otherwise still fail that coarser,
+ context-free check and collapse onto the same denied scope as a
+ genuinely unauthorized requester, letting the latter read the
+ former's cached payload. ``data_key`` is the annotation chart's own
+ query cache key(s) β derived from the same, override-applied query
+ objects actually executed (see :meth:`_apply_annotation_overrides`),
+ so it captures the datasource version, RLS clauses, and any per-user
+ Jinja/virtual-dataset RLS material exactly as the live fetch would,
+ including material an override only introduces at a finer grain.
+ Reusing this logic avoids re-deriving it and automatically inherits
+ any future correctness fixes made there.
+ """
+ layer_value = layer.get("value")
+ datasource = None
+ try:
chart = (
ChartDAO.find_by_id(layer_value) if layer_value is not None else None
)
- annotation_datasource = chart.datasource if chart else None
- source_rls[str(layer.get("value"))] = (
- security_manager.get_rls_cache_key(annotation_datasource)
- if annotation_datasource
- else None
+ # resolved_datasource, not datasource: the latter is pinned to
+ # table-backed datasources and resolves to None for a
+ # semantic-view-backed chart, which would otherwise collapse
+ # every requester onto the same {access: None, data_key: None}
+ # scope below regardless of their actual access.
+ datasource = chart.resolved_datasource if chart else None
+ if chart is None or datasource is None:
+ return {"access": None, "data_key": None}
+
+ annotation_query_context = chart.get_query_context()
+ if annotation_query_context is not None:
+ self._apply_annotation_overrides(annotation_query_context, layer)
+ try:
+ annotation_query_context.raise_for_access()
+ access: Any = True
+ except SupersetSecurityException:
+ access = False
+ data_key: Any = [
Review Comment:
For a saved annotation query without `time_range`, a relative override such as `1 hour ago : now` contributes newly resolved timestamps to this key on every request. That defeats annotation cache/nonce reuse and can rerun the same user's slow annotation query synchronously after an async task succeeds; should the overridden logical range be retained when deriving this scope?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] fix(charts): decouple annotation cache from the shared dataframe cache [superset]
Posted by "rusackas (via GitHub)" <gi...@apache.org>.
rusackas commented on code in PR #44406:
URL: https://github.com/apache/superset/pull/44406#discussion_r4172344448
##########
tests/unit_tests/common/test_query_context_processor.py:
##########
@@ -117,25 +118,259 @@ def processor(mock_query_context):
return processor
-def test_query_cache_key_binds_annotation_data_to_requesting_user(processor):
- """The cache key for annotated queries must differ per requesting user."""
+def test_annotation_cache_key_binds_native_annotation_read_scope(processor) -> None:
+ """The annotation cache key for NATIVE layers must differ when the
+ requester's ``can_read`` (Annotation) access differs -- not who they are."""
query_obj = MagicMock()
query_obj.annotation_layers = [{"sourceType": "NATIVE", "name": "a", "value": 1}]
- with (
- patch(
- "superset.common.query_context_processor.get_user_id",
- side_effect=[1, 2],
- ),
- patch("superset.common.query_context_processor.security_manager"),
- ):
- processor.query_cache_key(query_obj)
- processor.query_cache_key(query_obj)
+ # ``security_manager`` autodetects as an async spec under a bare
+ # ``patch()`` (its real object trips ``unittest.mock``'s coroutine
+ # inference), which would silently turn every attribute access into an
+ # ``AsyncMock`` returning a fresh unawaited coroutine per call -- always
+ # unequal to itself and never equal to a configured return value. Forcing
+ # ``new_callable=MagicMock`` keeps these synchronous, as the real object
+ # is.
+ with patch(
+ "superset.common.query_context_processor.security_manager",
+ new_callable=MagicMock,
+ ) as security_manager:
+ security_manager.can_access.side_effect = [True, False]
+ processor.annotation_cache_key(query_obj)
+ processor.annotation_cache_key(query_obj)
contexts = [
call.kwargs["annotation_context"] for call in query_obj.cache_key.call_args_list
]
assert contexts[0] != contexts[1]
+def test_annotation_cache_key_shares_across_same_access_scope() -> None:
+ """Two distinct requesters (separate processor/query-object instances,
+ standing in for two different requests) with identical access scope must
+ produce identical annotation-context material. Reusing a single
+ processor/query_obj across both calls (as this test previously did)
+ would pass trivially regardless of whether the key is scope-based or
+ identity-based, since nothing about "who's asking" would ever vary."""
+ layer = {"sourceType": "NATIVE", "name": "a", "value": 1}
+ processor_a = QueryContextProcessor(MagicMock())
+ processor_b = QueryContextProcessor(MagicMock())
+ query_obj_a = MagicMock(annotation_layers=[layer])
+ query_obj_b = MagicMock(annotation_layers=[layer])
+
+ with patch(
+ "superset.common.query_context_processor.security_manager",
+ new_callable=MagicMock,
+ ) as security_manager:
+ security_manager.can_access.return_value = True
+ context_a = processor_a._annotation_cache_context(query_obj_a)
+ context_b = processor_b._annotation_cache_context(query_obj_b)
+
+ assert context_a == context_b
+
+
+def test_query_cache_key_does_not_bind_annotation_scope(processor) -> None:
+ """The dataframe cache key must stay shared across viewers of the same
+ chart, even when the query has annotation layers β only the separate
+ annotation cache key (see above) carries access-scope material."""
+ query_obj = MagicMock()
+ query_obj.annotation_layers = [{"sourceType": "NATIVE", "name": "a", "value": 1}]
+ with patch(
+ "superset.common.query_context_processor.security_manager",
+ new_callable=MagicMock,
+ ):
+ processor.query_cache_key(query_obj)
+ processor.query_cache_key(query_obj)
+ for call in query_obj.cache_key.call_args_list:
+ assert "annotation_context" not in call.kwargs
+
+
+@pytest.fixture
+def mock_annotation_chart() -> Iterator[MagicMock]:
+ """A found chart, wired as the referenced chart for
+ ``_annotation_source_scope`` tests -- factors out the repeated
+ ``ChartDAO.find_by_id`` patch those tests all need."""
+ chart = MagicMock()
+ with patch(
+ "superset.common.query_context_processor.ChartDAO.find_by_id",
+ return_value=chart,
+ ):
+ yield chart
+
+
+def test_annotation_source_scope_binds_datasource_access(
+ processor, mock_annotation_chart
+) -> None:
+ """A chart-backed annotation layer's scope must differ when the
+ requester's access to the referenced datasource differs."""
+ mock_annotation_chart.get_query_context.return_value = None
+ with patch(
+ "superset.common.query_context_processor.security_manager",
+ new_callable=MagicMock,
+ ) as security_manager:
+ security_manager.can_access_datasource.side_effect = [True, False]
+ security_manager.get_rls_cache_key.return_value = []
+ scope_a = processor._annotation_source_scope({"value": 1})
+ scope_b = processor._annotation_source_scope({"value": 1})
+ assert scope_a != scope_b
+ assert scope_a["access"] is True
+ assert scope_b["access"] is False
+
+
+def test_annotation_source_scope_reuses_referenced_chart_cache_key(
+ processor, mock_annotation_chart
+) -> None:
Review Comment:
Fixed in the latest push.
##########
tests/unit_tests/common/test_query_context_processor.py:
##########
@@ -117,25 +118,259 @@ def processor(mock_query_context):
return processor
-def test_query_cache_key_binds_annotation_data_to_requesting_user(processor):
- """The cache key for annotated queries must differ per requesting user."""
+def test_annotation_cache_key_binds_native_annotation_read_scope(processor) -> None:
+ """The annotation cache key for NATIVE layers must differ when the
+ requester's ``can_read`` (Annotation) access differs -- not who they are."""
query_obj = MagicMock()
query_obj.annotation_layers = [{"sourceType": "NATIVE", "name": "a", "value": 1}]
- with (
- patch(
- "superset.common.query_context_processor.get_user_id",
- side_effect=[1, 2],
- ),
- patch("superset.common.query_context_processor.security_manager"),
- ):
- processor.query_cache_key(query_obj)
- processor.query_cache_key(query_obj)
+ # ``security_manager`` autodetects as an async spec under a bare
+ # ``patch()`` (its real object trips ``unittest.mock``'s coroutine
+ # inference), which would silently turn every attribute access into an
+ # ``AsyncMock`` returning a fresh unawaited coroutine per call -- always
+ # unequal to itself and never equal to a configured return value. Forcing
+ # ``new_callable=MagicMock`` keeps these synchronous, as the real object
+ # is.
+ with patch(
+ "superset.common.query_context_processor.security_manager",
+ new_callable=MagicMock,
+ ) as security_manager:
+ security_manager.can_access.side_effect = [True, False]
+ processor.annotation_cache_key(query_obj)
+ processor.annotation_cache_key(query_obj)
contexts = [
call.kwargs["annotation_context"] for call in query_obj.cache_key.call_args_list
]
assert contexts[0] != contexts[1]
+def test_annotation_cache_key_shares_across_same_access_scope() -> None:
+ """Two distinct requesters (separate processor/query-object instances,
+ standing in for two different requests) with identical access scope must
+ produce identical annotation-context material. Reusing a single
+ processor/query_obj across both calls (as this test previously did)
+ would pass trivially regardless of whether the key is scope-based or
+ identity-based, since nothing about "who's asking" would ever vary."""
+ layer = {"sourceType": "NATIVE", "name": "a", "value": 1}
+ processor_a = QueryContextProcessor(MagicMock())
+ processor_b = QueryContextProcessor(MagicMock())
+ query_obj_a = MagicMock(annotation_layers=[layer])
+ query_obj_b = MagicMock(annotation_layers=[layer])
+
+ with patch(
+ "superset.common.query_context_processor.security_manager",
+ new_callable=MagicMock,
+ ) as security_manager:
+ security_manager.can_access.return_value = True
+ context_a = processor_a._annotation_cache_context(query_obj_a)
+ context_b = processor_b._annotation_cache_context(query_obj_b)
+
+ assert context_a == context_b
+
+
+def test_query_cache_key_does_not_bind_annotation_scope(processor) -> None:
+ """The dataframe cache key must stay shared across viewers of the same
+ chart, even when the query has annotation layers β only the separate
+ annotation cache key (see above) carries access-scope material."""
+ query_obj = MagicMock()
+ query_obj.annotation_layers = [{"sourceType": "NATIVE", "name": "a", "value": 1}]
+ with patch(
+ "superset.common.query_context_processor.security_manager",
+ new_callable=MagicMock,
+ ):
+ processor.query_cache_key(query_obj)
+ processor.query_cache_key(query_obj)
+ for call in query_obj.cache_key.call_args_list:
+ assert "annotation_context" not in call.kwargs
+
+
+@pytest.fixture
+def mock_annotation_chart() -> Iterator[MagicMock]:
+ """A found chart, wired as the referenced chart for
+ ``_annotation_source_scope`` tests -- factors out the repeated
+ ``ChartDAO.find_by_id`` patch those tests all need."""
+ chart = MagicMock()
+ with patch(
+ "superset.common.query_context_processor.ChartDAO.find_by_id",
+ return_value=chart,
+ ):
+ yield chart
+
+
+def test_annotation_source_scope_binds_datasource_access(
+ processor, mock_annotation_chart
+) -> None:
+ """A chart-backed annotation layer's scope must differ when the
+ requester's access to the referenced datasource differs."""
+ mock_annotation_chart.get_query_context.return_value = None
+ with patch(
+ "superset.common.query_context_processor.security_manager",
+ new_callable=MagicMock,
+ ) as security_manager:
+ security_manager.can_access_datasource.side_effect = [True, False]
+ security_manager.get_rls_cache_key.return_value = []
+ scope_a = processor._annotation_source_scope({"value": 1})
+ scope_b = processor._annotation_source_scope({"value": 1})
+ assert scope_a != scope_b
+ assert scope_a["access"] is True
+ assert scope_b["access"] is False
+
+
+def test_annotation_source_scope_reuses_referenced_chart_cache_key(
+ processor, mock_annotation_chart
+) -> None:
+ """When the referenced chart has a saved query context, its own cache
+ key(s) -- covering RLS and per-user Jinja/virtual-dataset material -- are
+ reused rather than re-derived."""
+ mock_query_object = MagicMock()
+ mock_query_context = MagicMock()
Review Comment:
Fixed in the latest push.
##########
tests/unit_tests/common/test_query_context_processor.py:
##########
@@ -117,25 +118,259 @@ def processor(mock_query_context):
return processor
-def test_query_cache_key_binds_annotation_data_to_requesting_user(processor):
- """The cache key for annotated queries must differ per requesting user."""
+def test_annotation_cache_key_binds_native_annotation_read_scope(processor) -> None:
+ """The annotation cache key for NATIVE layers must differ when the
+ requester's ``can_read`` (Annotation) access differs -- not who they are."""
query_obj = MagicMock()
query_obj.annotation_layers = [{"sourceType": "NATIVE", "name": "a", "value": 1}]
- with (
- patch(
- "superset.common.query_context_processor.get_user_id",
- side_effect=[1, 2],
- ),
- patch("superset.common.query_context_processor.security_manager"),
- ):
- processor.query_cache_key(query_obj)
- processor.query_cache_key(query_obj)
+ # ``security_manager`` autodetects as an async spec under a bare
+ # ``patch()`` (its real object trips ``unittest.mock``'s coroutine
+ # inference), which would silently turn every attribute access into an
+ # ``AsyncMock`` returning a fresh unawaited coroutine per call -- always
+ # unequal to itself and never equal to a configured return value. Forcing
+ # ``new_callable=MagicMock`` keeps these synchronous, as the real object
+ # is.
+ with patch(
+ "superset.common.query_context_processor.security_manager",
+ new_callable=MagicMock,
+ ) as security_manager:
+ security_manager.can_access.side_effect = [True, False]
+ processor.annotation_cache_key(query_obj)
+ processor.annotation_cache_key(query_obj)
contexts = [
call.kwargs["annotation_context"] for call in query_obj.cache_key.call_args_list
]
assert contexts[0] != contexts[1]
+def test_annotation_cache_key_shares_across_same_access_scope() -> None:
+ """Two distinct requesters (separate processor/query-object instances,
+ standing in for two different requests) with identical access scope must
+ produce identical annotation-context material. Reusing a single
+ processor/query_obj across both calls (as this test previously did)
+ would pass trivially regardless of whether the key is scope-based or
+ identity-based, since nothing about "who's asking" would ever vary."""
+ layer = {"sourceType": "NATIVE", "name": "a", "value": 1}
+ processor_a = QueryContextProcessor(MagicMock())
+ processor_b = QueryContextProcessor(MagicMock())
+ query_obj_a = MagicMock(annotation_layers=[layer])
+ query_obj_b = MagicMock(annotation_layers=[layer])
+
+ with patch(
+ "superset.common.query_context_processor.security_manager",
+ new_callable=MagicMock,
+ ) as security_manager:
+ security_manager.can_access.return_value = True
+ context_a = processor_a._annotation_cache_context(query_obj_a)
+ context_b = processor_b._annotation_cache_context(query_obj_b)
+
+ assert context_a == context_b
+
+
+def test_query_cache_key_does_not_bind_annotation_scope(processor) -> None:
+ """The dataframe cache key must stay shared across viewers of the same
+ chart, even when the query has annotation layers β only the separate
+ annotation cache key (see above) carries access-scope material."""
+ query_obj = MagicMock()
+ query_obj.annotation_layers = [{"sourceType": "NATIVE", "name": "a", "value": 1}]
+ with patch(
+ "superset.common.query_context_processor.security_manager",
+ new_callable=MagicMock,
+ ):
+ processor.query_cache_key(query_obj)
+ processor.query_cache_key(query_obj)
+ for call in query_obj.cache_key.call_args_list:
+ assert "annotation_context" not in call.kwargs
+
+
+@pytest.fixture
+def mock_annotation_chart() -> Iterator[MagicMock]:
+ """A found chart, wired as the referenced chart for
+ ``_annotation_source_scope`` tests -- factors out the repeated
+ ``ChartDAO.find_by_id`` patch those tests all need."""
+ chart = MagicMock()
+ with patch(
+ "superset.common.query_context_processor.ChartDAO.find_by_id",
+ return_value=chart,
+ ):
+ yield chart
+
+
+def test_annotation_source_scope_binds_datasource_access(
+ processor, mock_annotation_chart
+) -> None:
+ """A chart-backed annotation layer's scope must differ when the
+ requester's access to the referenced datasource differs."""
+ mock_annotation_chart.get_query_context.return_value = None
+ with patch(
+ "superset.common.query_context_processor.security_manager",
+ new_callable=MagicMock,
+ ) as security_manager:
+ security_manager.can_access_datasource.side_effect = [True, False]
+ security_manager.get_rls_cache_key.return_value = []
+ scope_a = processor._annotation_source_scope({"value": 1})
+ scope_b = processor._annotation_source_scope({"value": 1})
+ assert scope_a != scope_b
+ assert scope_a["access"] is True
+ assert scope_b["access"] is False
+
+
+def test_annotation_source_scope_reuses_referenced_chart_cache_key(
+ processor, mock_annotation_chart
+) -> None:
+ """When the referenced chart has a saved query context, its own cache
+ key(s) -- covering RLS and per-user Jinja/virtual-dataset material -- are
+ reused rather than re-derived."""
+ mock_query_object = MagicMock()
+ mock_query_context = MagicMock()
+ mock_query_context.queries = [mock_query_object]
+ mock_query_context.query_cache_key.return_value = "referenced-chart-key"
+ mock_annotation_chart.get_query_context.return_value = mock_query_context
+ with patch(
+ "superset.common.query_context_processor.security_manager",
+ new_callable=MagicMock,
+ ) as security_manager:
+ security_manager.can_access_datasource.return_value = True
+ scope = processor._annotation_source_scope({"value": 1})
+ assert scope == {"access": True, "data_key": ["referenced-chart-key"]}
+ mock_query_context.query_cache_key.assert_called_once_with(mock_query_object)
+
+
+def test_annotation_source_scope_uses_live_fetch_authorization(
+ processor, mock_annotation_chart
+) -> None:
Review Comment:
Fixed in the latest push.
##########
tests/unit_tests/common/test_query_context_processor.py:
##########
@@ -117,25 +118,259 @@ def processor(mock_query_context):
return processor
-def test_query_cache_key_binds_annotation_data_to_requesting_user(processor):
- """The cache key for annotated queries must differ per requesting user."""
+def test_annotation_cache_key_binds_native_annotation_read_scope(processor) -> None:
+ """The annotation cache key for NATIVE layers must differ when the
+ requester's ``can_read`` (Annotation) access differs -- not who they are."""
query_obj = MagicMock()
query_obj.annotation_layers = [{"sourceType": "NATIVE", "name": "a", "value": 1}]
- with (
- patch(
- "superset.common.query_context_processor.get_user_id",
- side_effect=[1, 2],
- ),
- patch("superset.common.query_context_processor.security_manager"),
- ):
- processor.query_cache_key(query_obj)
- processor.query_cache_key(query_obj)
+ # ``security_manager`` autodetects as an async spec under a bare
+ # ``patch()`` (its real object trips ``unittest.mock``'s coroutine
+ # inference), which would silently turn every attribute access into an
+ # ``AsyncMock`` returning a fresh unawaited coroutine per call -- always
+ # unequal to itself and never equal to a configured return value. Forcing
+ # ``new_callable=MagicMock`` keeps these synchronous, as the real object
+ # is.
+ with patch(
+ "superset.common.query_context_processor.security_manager",
+ new_callable=MagicMock,
+ ) as security_manager:
+ security_manager.can_access.side_effect = [True, False]
+ processor.annotation_cache_key(query_obj)
+ processor.annotation_cache_key(query_obj)
contexts = [
call.kwargs["annotation_context"] for call in query_obj.cache_key.call_args_list
]
assert contexts[0] != contexts[1]
+def test_annotation_cache_key_shares_across_same_access_scope() -> None:
+ """Two distinct requesters (separate processor/query-object instances,
+ standing in for two different requests) with identical access scope must
+ produce identical annotation-context material. Reusing a single
+ processor/query_obj across both calls (as this test previously did)
+ would pass trivially regardless of whether the key is scope-based or
+ identity-based, since nothing about "who's asking" would ever vary."""
+ layer = {"sourceType": "NATIVE", "name": "a", "value": 1}
+ processor_a = QueryContextProcessor(MagicMock())
+ processor_b = QueryContextProcessor(MagicMock())
+ query_obj_a = MagicMock(annotation_layers=[layer])
+ query_obj_b = MagicMock(annotation_layers=[layer])
+
+ with patch(
+ "superset.common.query_context_processor.security_manager",
+ new_callable=MagicMock,
+ ) as security_manager:
+ security_manager.can_access.return_value = True
+ context_a = processor_a._annotation_cache_context(query_obj_a)
+ context_b = processor_b._annotation_cache_context(query_obj_b)
+
+ assert context_a == context_b
+
+
+def test_query_cache_key_does_not_bind_annotation_scope(processor) -> None:
+ """The dataframe cache key must stay shared across viewers of the same
+ chart, even when the query has annotation layers β only the separate
+ annotation cache key (see above) carries access-scope material."""
+ query_obj = MagicMock()
+ query_obj.annotation_layers = [{"sourceType": "NATIVE", "name": "a", "value": 1}]
+ with patch(
+ "superset.common.query_context_processor.security_manager",
+ new_callable=MagicMock,
+ ):
+ processor.query_cache_key(query_obj)
+ processor.query_cache_key(query_obj)
+ for call in query_obj.cache_key.call_args_list:
+ assert "annotation_context" not in call.kwargs
+
+
+@pytest.fixture
+def mock_annotation_chart() -> Iterator[MagicMock]:
+ """A found chart, wired as the referenced chart for
+ ``_annotation_source_scope`` tests -- factors out the repeated
+ ``ChartDAO.find_by_id`` patch those tests all need."""
+ chart = MagicMock()
+ with patch(
+ "superset.common.query_context_processor.ChartDAO.find_by_id",
+ return_value=chart,
+ ):
+ yield chart
+
+
+def test_annotation_source_scope_binds_datasource_access(
+ processor, mock_annotation_chart
+) -> None:
+ """A chart-backed annotation layer's scope must differ when the
+ requester's access to the referenced datasource differs."""
+ mock_annotation_chart.get_query_context.return_value = None
+ with patch(
+ "superset.common.query_context_processor.security_manager",
+ new_callable=MagicMock,
+ ) as security_manager:
+ security_manager.can_access_datasource.side_effect = [True, False]
+ security_manager.get_rls_cache_key.return_value = []
+ scope_a = processor._annotation_source_scope({"value": 1})
+ scope_b = processor._annotation_source_scope({"value": 1})
+ assert scope_a != scope_b
+ assert scope_a["access"] is True
+ assert scope_b["access"] is False
+
+
+def test_annotation_source_scope_reuses_referenced_chart_cache_key(
+ processor, mock_annotation_chart
+) -> None:
+ """When the referenced chart has a saved query context, its own cache
+ key(s) -- covering RLS and per-user Jinja/virtual-dataset material -- are
+ reused rather than re-derived."""
+ mock_query_object = MagicMock()
+ mock_query_context = MagicMock()
+ mock_query_context.queries = [mock_query_object]
+ mock_query_context.query_cache_key.return_value = "referenced-chart-key"
+ mock_annotation_chart.get_query_context.return_value = mock_query_context
+ with patch(
+ "superset.common.query_context_processor.security_manager",
+ new_callable=MagicMock,
+ ) as security_manager:
+ security_manager.can_access_datasource.return_value = True
+ scope = processor._annotation_source_scope({"value": 1})
+ assert scope == {"access": True, "data_key": ["referenced-chart-key"]}
+ mock_query_context.query_cache_key.assert_called_once_with(mock_query_object)
+
+
+def test_annotation_source_scope_uses_live_fetch_authorization(
+ processor, mock_annotation_chart
+) -> None:
+ """When the referenced chart has a saved query context, ``access`` must
+ come from that context's own ``raise_for_access`` -- the same
+ authorization path the live fetch in ``get_viz_annotation_data`` uses --
+ not the coarser, context-free ``can_access_datasource``. A requester
+ denied by ``can_access_datasource`` but granted via a bypass that depends
+ on the chart's own saved form_data (e.g. a dashboard/viewer-promiscuous
+ bypass) must get a scope distinct from one truly denied, or the latter
+ could read the former's cached payload."""
+ from superset.exceptions import SupersetSecurityException
Review Comment:
Fixed in the latest push.
##########
tests/unit_tests/common/test_query_context_processor.py:
##########
@@ -117,25 +118,259 @@ def processor(mock_query_context):
return processor
-def test_query_cache_key_binds_annotation_data_to_requesting_user(processor):
- """The cache key for annotated queries must differ per requesting user."""
+def test_annotation_cache_key_binds_native_annotation_read_scope(processor) -> None:
+ """The annotation cache key for NATIVE layers must differ when the
+ requester's ``can_read`` (Annotation) access differs -- not who they are."""
query_obj = MagicMock()
query_obj.annotation_layers = [{"sourceType": "NATIVE", "name": "a", "value": 1}]
- with (
- patch(
- "superset.common.query_context_processor.get_user_id",
- side_effect=[1, 2],
- ),
- patch("superset.common.query_context_processor.security_manager"),
- ):
- processor.query_cache_key(query_obj)
- processor.query_cache_key(query_obj)
+ # ``security_manager`` autodetects as an async spec under a bare
+ # ``patch()`` (its real object trips ``unittest.mock``'s coroutine
+ # inference), which would silently turn every attribute access into an
+ # ``AsyncMock`` returning a fresh unawaited coroutine per call -- always
+ # unequal to itself and never equal to a configured return value. Forcing
+ # ``new_callable=MagicMock`` keeps these synchronous, as the real object
+ # is.
+ with patch(
+ "superset.common.query_context_processor.security_manager",
+ new_callable=MagicMock,
+ ) as security_manager:
+ security_manager.can_access.side_effect = [True, False]
+ processor.annotation_cache_key(query_obj)
+ processor.annotation_cache_key(query_obj)
contexts = [
call.kwargs["annotation_context"] for call in query_obj.cache_key.call_args_list
]
assert contexts[0] != contexts[1]
+def test_annotation_cache_key_shares_across_same_access_scope() -> None:
+ """Two distinct requesters (separate processor/query-object instances,
+ standing in for two different requests) with identical access scope must
+ produce identical annotation-context material. Reusing a single
+ processor/query_obj across both calls (as this test previously did)
+ would pass trivially regardless of whether the key is scope-based or
+ identity-based, since nothing about "who's asking" would ever vary."""
+ layer = {"sourceType": "NATIVE", "name": "a", "value": 1}
+ processor_a = QueryContextProcessor(MagicMock())
+ processor_b = QueryContextProcessor(MagicMock())
+ query_obj_a = MagicMock(annotation_layers=[layer])
+ query_obj_b = MagicMock(annotation_layers=[layer])
+
+ with patch(
+ "superset.common.query_context_processor.security_manager",
+ new_callable=MagicMock,
+ ) as security_manager:
+ security_manager.can_access.return_value = True
+ context_a = processor_a._annotation_cache_context(query_obj_a)
+ context_b = processor_b._annotation_cache_context(query_obj_b)
+
+ assert context_a == context_b
+
+
+def test_query_cache_key_does_not_bind_annotation_scope(processor) -> None:
+ """The dataframe cache key must stay shared across viewers of the same
+ chart, even when the query has annotation layers β only the separate
+ annotation cache key (see above) carries access-scope material."""
+ query_obj = MagicMock()
+ query_obj.annotation_layers = [{"sourceType": "NATIVE", "name": "a", "value": 1}]
+ with patch(
+ "superset.common.query_context_processor.security_manager",
+ new_callable=MagicMock,
+ ):
+ processor.query_cache_key(query_obj)
+ processor.query_cache_key(query_obj)
+ for call in query_obj.cache_key.call_args_list:
+ assert "annotation_context" not in call.kwargs
+
+
+@pytest.fixture
+def mock_annotation_chart() -> Iterator[MagicMock]:
+ """A found chart, wired as the referenced chart for
+ ``_annotation_source_scope`` tests -- factors out the repeated
+ ``ChartDAO.find_by_id`` patch those tests all need."""
+ chart = MagicMock()
+ with patch(
+ "superset.common.query_context_processor.ChartDAO.find_by_id",
+ return_value=chart,
+ ):
+ yield chart
+
+
+def test_annotation_source_scope_binds_datasource_access(
+ processor, mock_annotation_chart
+) -> None:
+ """A chart-backed annotation layer's scope must differ when the
+ requester's access to the referenced datasource differs."""
+ mock_annotation_chart.get_query_context.return_value = None
+ with patch(
+ "superset.common.query_context_processor.security_manager",
+ new_callable=MagicMock,
+ ) as security_manager:
+ security_manager.can_access_datasource.side_effect = [True, False]
+ security_manager.get_rls_cache_key.return_value = []
+ scope_a = processor._annotation_source_scope({"value": 1})
+ scope_b = processor._annotation_source_scope({"value": 1})
+ assert scope_a != scope_b
+ assert scope_a["access"] is True
+ assert scope_b["access"] is False
+
+
+def test_annotation_source_scope_reuses_referenced_chart_cache_key(
+ processor, mock_annotation_chart
+) -> None:
+ """When the referenced chart has a saved query context, its own cache
+ key(s) -- covering RLS and per-user Jinja/virtual-dataset material -- are
+ reused rather than re-derived."""
+ mock_query_object = MagicMock()
+ mock_query_context = MagicMock()
+ mock_query_context.queries = [mock_query_object]
+ mock_query_context.query_cache_key.return_value = "referenced-chart-key"
+ mock_annotation_chart.get_query_context.return_value = mock_query_context
+ with patch(
+ "superset.common.query_context_processor.security_manager",
+ new_callable=MagicMock,
+ ) as security_manager:
+ security_manager.can_access_datasource.return_value = True
+ scope = processor._annotation_source_scope({"value": 1})
+ assert scope == {"access": True, "data_key": ["referenced-chart-key"]}
+ mock_query_context.query_cache_key.assert_called_once_with(mock_query_object)
+
+
+def test_annotation_source_scope_uses_live_fetch_authorization(
+ processor, mock_annotation_chart
+) -> None:
+ """When the referenced chart has a saved query context, ``access`` must
+ come from that context's own ``raise_for_access`` -- the same
+ authorization path the live fetch in ``get_viz_annotation_data`` uses --
+ not the coarser, context-free ``can_access_datasource``. A requester
+ denied by ``can_access_datasource`` but granted via a bypass that depends
+ on the chart's own saved form_data (e.g. a dashboard/viewer-promiscuous
+ bypass) must get a scope distinct from one truly denied, or the latter
+ could read the former's cached payload."""
+ from superset.exceptions import SupersetSecurityException
+
+ mock_query_object = MagicMock()
+ mock_query_context = MagicMock()
Review Comment:
Fixed in the latest push.
##########
tests/unit_tests/common/test_query_context_processor.py:
##########
@@ -2630,6 +3041,94 @@ def test_mark_force_executed_noop_without_nonce(processor, mock_query_context):
cache_manager.data_cache.set.assert_not_called()
+# =============================================================================
+# Annotation-data cache decoupled from the dataframe cache
+# =============================================================================
+
+
+def test_get_annotation_data_cached_reads_from_cache(processor):
Review Comment:
Fixed in the latest push.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] fix(charts): decouple annotation cache from the shared dataframe cache [superset]
Posted by "bito-code-review[bot] (via GitHub)" <gi...@apache.org>.
bito-code-review[bot] commented on PR #44406:
URL: https://github.com/apache/superset/pull/44406#issuecomment-5890089997
AI Code Review is in progress (usually takes 3 to 15 minutes unless it's a very large PR).
<details>
<summary>
Bito Usage Guide
</summary>
**Commands**
Type the following command in the pull request comment and save the comment.
- `/review` - Manually triggers an incremental AI Review.
- `/review full` - Manually triggers a full AI Review.
- `/pause` - Pauses automatic reviews on this pull request.
- `/resume` - Resumes automatic reviews.
- `/resolve` - Marks all Bito-posted review comments as resolved.
- `/abort` - Cancels all in-progress reviews.
Refer to the <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/available-commands" target="_blank" rel="noopener noreferrer">documentation</a> for additional commands.
**Configuration**
This repository uses `Superset` You can customize the agent settings <a href="https://alpha.bito.ai/home/ai-agents/code-review-agent" target="_blank" rel="noopener noreferrer">here</a> or contact your Bito workspace admin at evan@preset.io.
**Documentation & Help**
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/getting-started/install-run-using-bito-cloud/create-or-customize-an-agent-instance" target="_blank" rel="noopener noreferrer">Customize agent settings</a>
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/implementing-custom-code-review-rules" target="_blank" rel="noopener noreferrer">Review rules</a>
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent" target="_blank" rel="noopener noreferrer">General documentation</a>
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/faqs" target="_blank" rel="noopener noreferrer">FAQ</a>
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] fix(charts): decouple annotation cache from the shared dataframe cache [superset]
Posted by "bito-code-review[bot] (via GitHub)" <gi...@apache.org>.
bito-code-review[bot] commented on code in PR #44406:
URL: https://github.com/apache/superset/pull/44406#discussion_r4139022326
##########
tests/unit_tests/common/test_query_context_processor.py:
##########
@@ -2226,6 +2396,180 @@ def test_get_df_payload_no_warning_when_not_memory_limited() -> None:
assert result["warning"] is None
+def test_get_df_payload_result_decouples_annotation_cache_from_dataframe_cache():
Review Comment:
<!-- Bito Reply -->
The addition of the return type hint is appropriate and aligns with the project's coding standards for test methods. This change ensures consistency with existing tests and supports static type checking.
**tests/unit_tests/common/test_query_context_processor.py**
```
def test_get_df_payload_result_decouples_annotation_cache_from_dataframe_cache() -> None:
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] fix(charts): decouple annotation cache from the shared dataframe cache [superset]
Posted by "rusackas (via GitHub)" <gi...@apache.org>.
rusackas commented on code in PR #44406:
URL: https://github.com/apache/superset/pull/44406#discussion_r4332207793
##########
tests/unit_tests/tasks/test_query_cancel.py:
##########
@@ -159,6 +159,26 @@ def test_capture_registers_abort_handler_once_when_id_available() -> None:
assert ordered.index("set_cancellation") < ordered.index("on_abort")
+def test_capture_ignores_cursor_from_a_different_database() -> None:
Review Comment:
Added a test that goes through `notify_cursor` itself: the task's own database registers the abort handler once across two cursors, and a different database registers nothing.
##########
superset/common/query_context_processor.py:
##########
@@ -285,6 +279,18 @@ def get_df_payload_result(
# decision as the result cache, not the datasource-level default.
query_obj.force_query = force_query
query_obj.cache_timeout = timeout
+ # Resolved separately from the dataframe's force_query, against its own
+ # annotation_key. A forced GTF refresh's nonce marker is per (nonce,
+ # cache_key): the dataframe's marker is keyed on cache_key, so once it's
+ # set, force_query goes False for every subsequent request carrying that
+ # nonce, including a request for a *different* annotation access scope
+ # (its own annotation_key) that has never actually forced-refreshed its
+ # own entry. Reusing the dataframe's force_query there would read that
+ # scope's existing annotation cache entry instead of refreshing it.
+ annotation_force_query = (
+ self._resolve_forced_query(query_obj, annotation_key)
+ or timeout == CACHE_DISABLED_TIMEOUT
Review Comment:
Added a `get_df_payload_result` case with `get_cache_timeout=-1`, `force=False` and a warm annotation entry: the cache is never read and the live fetch's result is what lands in the payload.
##########
tests/unit_tests/common/test_query_context_processor.py:
##########
@@ -3167,11 +3915,211 @@ def payload(context: QueryContext, **kwargs: Any) -> dict[str, Any]:
datasource="native-1", annotation_context=context
)
assert current_key != legacy_key
- assert context == {
- **legacy_context,
- "semantic_filter_protocol": "semantic-null-filters-v1",
- "source_versions": {"7": view.metadata_generation},
- }
+ assert context is not None
+ assert context["source_scope"][0]["semantic_filter_protocol"] == (
+ "semantic-null-filters-v1"
+ )
+ assert context["source_scope"][0]["metadata_version"] == view.metadata_generation
+
+
+def _annotated_payload_fixture(
+ *, status: str = "success"
+) -> tuple[QueryContextProcessor, Any, MagicMock]:
+ """A processor, annotated query object, and a dataframe cache hit."""
+ from superset.common.query_object import QueryObject
+
+ mock_query_context = MagicMock()
+ mock_query_context.force = False
+ mock_datasource = MagicMock()
+ mock_datasource.column_names = ["col1"]
+ processor = QueryContextProcessor(mock_query_context)
+ processor._qc_datasource = mock_datasource
+ query_obj = QueryObject(
+ datasource=mock_datasource,
+ columns=["col1"],
+ annotation_layers=[
+ {
+ "annotationType": "EVENT",
+ "sourceType": "NATIVE",
+ "name": "a",
+ "value": 1,
+ }
+ ],
+ )
+ cache = MagicMock()
+ cache.is_loaded = True
+ cache.applied_filter_columns = ["col1"]
+ cache.df = pd.DataFrame({"col1": [1, 2, 3]})
+ cache.query = ""
+ cache.status = status
+ cache.error_message = "dataframe failed" if status == QueryStatus.FAILED else None
+ cache.is_cached = True
+ cache.applied_template_filters = []
+ cache.rejected_filter_columns = []
+ cache.bq_memory_limited = False
+ cache.result_persisted = False
+ return processor, query_obj, cache
+
+
+def _run_annotated_payload(
+ processor: QueryContextProcessor,
+ query_obj: Any,
+ cache: MagicMock,
+ *,
+ force_cached: bool = False,
+) -> Any:
+ with (
+ patch(
+ "superset.common.query_context_processor.QueryCacheManager"
+ ) as mock_cache_manager,
+ patch.object(query_obj, "validate", return_value=None),
+ patch.object(processor, "query_cache_key", return_value="df-key"),
+ patch.object(processor, "annotation_cache_key", return_value="ann-key"),
+ patch.object(processor, "get_cache_timeout", return_value=3600),
+ ):
+ mock_cache_manager.get.return_value = cache
+ return processor.get_df_payload_result(query_obj, force_cached=force_cached)
+
+
+def test_get_df_payload_result_skips_annotations_when_dataframe_failed() -> None:
+ """A failed dataframe keeps its own error and never triggers the fetch."""
+ processor, query_obj, cache = _annotated_payload_fixture(status=QueryStatus.FAILED)
+ with patch.object(processor, "_get_annotation_data_cached") as mock_annotation:
+ result = _run_annotated_payload(processor, query_obj, cache)
+ mock_annotation.assert_not_called()
+ assert result.payload["annotation_data"] == {}
+ assert result.payload["error"] == "dataframe failed"
+
+
+def test_get_df_payload_result_force_cached_misses_without_annotation_entry() -> None:
+ """Through the real helper, ``force_cached`` with no annotation entry
+ raises ``CacheLoadError`` instead of running the annotation SQL."""
+ processor, query_obj, cache = _annotated_payload_fixture()
+ with (
+ patch("superset.common.query_context_processor.cache_manager") as cm,
+ patch.object(processor, "get_annotation_data") as mock_live,
+ ):
+ cm.data_cache.get.return_value = None
+ with pytest.raises(CacheLoadError):
+ _run_annotated_payload(processor, query_obj, cache, force_cached=True)
+ mock_live.assert_not_called()
+
+
+def test_get_df_payload_result_annotation_failure_marks_payload_failed() -> None:
+ """An annotation error on a good dataframe fails the payload."""
+ processor, query_obj, cache = _annotated_payload_fixture()
+ with (
+ patch("superset.common.query_context_processor.cache_manager") as cm,
+ patch.object(
+ processor,
+ "get_annotation_data",
+ side_effect=QueryObjectValidationError("annotation boom"),
+ ),
+ ):
+ cm.data_cache.get.return_value = None
+ result = _run_annotated_payload(processor, query_obj, cache)
+ assert cache.status == QueryStatus.FAILED
+ assert result.payload["error"] == "annotation boom"
Review Comment:
Good catch. That test now asserts `isinstance(..., QueryObjectValidationMessage)` and that `.cause` is the original exception.
##########
tests/unit_tests/common/test_query_context_processor.py:
##########
@@ -3167,11 +3915,211 @@ def payload(context: QueryContext, **kwargs: Any) -> dict[str, Any]:
datasource="native-1", annotation_context=context
)
assert current_key != legacy_key
- assert context == {
- **legacy_context,
- "semantic_filter_protocol": "semantic-null-filters-v1",
- "source_versions": {"7": view.metadata_generation},
- }
+ assert context is not None
+ assert context["source_scope"][0]["semantic_filter_protocol"] == (
+ "semantic-null-filters-v1"
+ )
+ assert context["source_scope"][0]["metadata_version"] == view.metadata_generation
+
+
+def _annotated_payload_fixture(
+ *, status: str = "success"
+) -> tuple[QueryContextProcessor, Any, MagicMock]:
+ """A processor, annotated query object, and a dataframe cache hit."""
+ from superset.common.query_object import QueryObject
+
+ mock_query_context = MagicMock()
+ mock_query_context.force = False
+ mock_datasource = MagicMock()
+ mock_datasource.column_names = ["col1"]
+ processor = QueryContextProcessor(mock_query_context)
+ processor._qc_datasource = mock_datasource
+ query_obj = QueryObject(
+ datasource=mock_datasource,
+ columns=["col1"],
+ annotation_layers=[
+ {
+ "annotationType": "EVENT",
+ "sourceType": "NATIVE",
+ "name": "a",
+ "value": 1,
+ }
+ ],
+ )
+ cache = MagicMock()
+ cache.is_loaded = True
+ cache.applied_filter_columns = ["col1"]
+ cache.df = pd.DataFrame({"col1": [1, 2, 3]})
+ cache.query = ""
+ cache.status = status
+ cache.error_message = "dataframe failed" if status == QueryStatus.FAILED else None
+ cache.is_cached = True
+ cache.applied_template_filters = []
+ cache.rejected_filter_columns = []
+ cache.bq_memory_limited = False
+ cache.result_persisted = False
+ return processor, query_obj, cache
+
+
+def _run_annotated_payload(
+ processor: QueryContextProcessor,
+ query_obj: Any,
+ cache: MagicMock,
+ *,
+ force_cached: bool = False,
+) -> Any:
+ with (
+ patch(
+ "superset.common.query_context_processor.QueryCacheManager"
+ ) as mock_cache_manager,
+ patch.object(query_obj, "validate", return_value=None),
+ patch.object(processor, "query_cache_key", return_value="df-key"),
+ patch.object(processor, "annotation_cache_key", return_value="ann-key"),
+ patch.object(processor, "get_cache_timeout", return_value=3600),
+ ):
+ mock_cache_manager.get.return_value = cache
+ return processor.get_df_payload_result(query_obj, force_cached=force_cached)
+
+
+def test_get_df_payload_result_skips_annotations_when_dataframe_failed() -> None:
+ """A failed dataframe keeps its own error and never triggers the fetch."""
+ processor, query_obj, cache = _annotated_payload_fixture(status=QueryStatus.FAILED)
+ with patch.object(processor, "_get_annotation_data_cached") as mock_annotation:
+ result = _run_annotated_payload(processor, query_obj, cache)
+ mock_annotation.assert_not_called()
+ assert result.payload["annotation_data"] == {}
+ assert result.payload["error"] == "dataframe failed"
+
+
+def test_get_df_payload_result_force_cached_misses_without_annotation_entry() -> None:
+ """Through the real helper, ``force_cached`` with no annotation entry
+ raises ``CacheLoadError`` instead of running the annotation SQL."""
+ processor, query_obj, cache = _annotated_payload_fixture()
+ with (
+ patch("superset.common.query_context_processor.cache_manager") as cm,
+ patch.object(processor, "get_annotation_data") as mock_live,
+ ):
+ cm.data_cache.get.return_value = None
+ with pytest.raises(CacheLoadError):
+ _run_annotated_payload(processor, query_obj, cache, force_cached=True)
+ mock_live.assert_not_called()
+
+
+def test_get_df_payload_result_annotation_failure_marks_payload_failed() -> None:
+ """An annotation error on a good dataframe fails the payload."""
+ processor, query_obj, cache = _annotated_payload_fixture()
+ with (
+ patch("superset.common.query_context_processor.cache_manager") as cm,
+ patch.object(
+ processor,
+ "get_annotation_data",
+ side_effect=QueryObjectValidationError("annotation boom"),
+ ),
+ ):
+ cm.data_cache.get.return_value = None
+ result = _run_annotated_payload(processor, query_obj, cache)
+ assert cache.status == QueryStatus.FAILED
+ assert result.payload["error"] == "annotation boom"
+
+
+def test_get_df_payload_result_completeness_error_escapes_annotation_fetch() -> None:
+ """A semantic annotation's incomplete-result signal must propagate (as the
+ dataframe path does) so the async task is not reported successful."""
+ from superset.exceptions import SemanticResultCompletenessError
+
+ processor, query_obj, cache = _annotated_payload_fixture()
+ with (
+ patch("superset.common.query_context_processor.cache_manager") as cm,
+ patch.object(
+ processor,
+ "get_annotation_data",
+ side_effect=SemanticResultCompletenessError("unverified"),
+ ),
+ ):
+ cm.data_cache.get.return_value = None
+ with pytest.raises(SemanticResultCompletenessError):
+ _run_annotated_payload(processor, query_obj, cache)
+
+
+def test_get_df_payload_result_times_annotation_fetch_as_data_acquisition() -> None:
+ """Annotation work lands in ``data_acquisition_ns`` even on a dataframe
+ cache hit (which otherwise reports no acquisition time)."""
+ processor, query_obj, cache = _annotated_payload_fixture()
+
+ def slow_annotations(*_: Any, **__: Any) -> dict[str, Any]:
+ time.sleep(0.01)
+ return {}
+
+ with patch.object(
+ processor, "_get_annotation_data_cached", side_effect=slow_annotations
+ ):
+ result = _run_annotated_payload(processor, query_obj, cache)
+ assert result.timing.data_acquisition_ns is not None
+ assert result.timing.data_acquisition_ns >= 10_000_000
Review Comment:
Added a deterministic-clock case on a dataframe miss: the dataframe fetch advances 30 ns, the annotation fetch 20 ns, and it asserts `data_acquisition_ns == 50`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] fix(charts): decouple annotation cache from the shared dataframe cache [superset]
Posted by "rusackas (via GitHub)" <gi...@apache.org>.
rusackas commented on code in PR #44406:
URL: https://github.com/apache/superset/pull/44406#discussion_r4332206832
##########
superset/common/query_context_processor.py:
##########
@@ -449,34 +481,189 @@ def query_cache_key(self, query_obj: QueryObject, **kwargs: Any) -> str | None:
)
return cache_key
- def _annotation_cache_context(self, query_obj: QueryObject) -> dict[str, Any]:
+ def annotation_cache_key(self, query_obj: QueryObject) -> str | None:
+ """
+ Cache key for this query's annotation-layer payload, or ``None`` when
+ the query has no annotation layers.
+
+ Annotation payloads are fetched under the requesting user's access
+ scope, which is a stricter security requirement than the dataframe
+ itself has. Keying them separately from :meth:`query_cache_key` keeps
+ that scoping from forcing every distinct viewer of an annotated chart
+ onto their own full copy of the (potentially much larger) shared
+ dataframe: users with the same access scope share this key too.
"""
- Cache-key material binding cached annotation data to its security
- context.
+ if not query_obj or not query_obj.annotation_layers:
+ return None
+ return self.query_cache_key(
+ query_obj, annotation_context=self._annotation_cache_context(query_obj)
+ )
- Annotation payloads are fetched per requesting user and stored on the
- same cache entry as the dataframe, so the key also binds the requesting
- user and, for chart-backed layers, the RLS clauses of the referenced
- chart's datasource.
+ def _annotation_cache_context(self, query_obj: QueryObject) -> dict[str, Any]:
+ """
+ Cache-key material binding annotation data to its security *scope* so
+ users with the same access share a cache entry and users with a
+ different scope β or no access β never read each other's data.
+
+ * NATIVE layers: the ``can_read`` permission on ``Annotation``, the
+ only user-dependent dimension of these global records.
+ * Chart-backed (``line``/``table``) layers: see
+ :meth:`_annotation_source_scope`.
"""
- source_rls: dict[str, list[str] | None] = {}
+ context: dict[str, Any] = {}
+
+ if any(
+ layer.get("sourceType") == "NATIVE" for layer in query_obj.annotation_layers
+ ):
+ context["annotation_read"] = security_manager.can_access(
+ "can_read", "Annotation"
+ )
+
+ source_scope: dict[str, Any] = {}
for layer in query_obj.annotation_layers:
if (
layer.get("sourceType")
not in ANNOTATION_SOURCE_TYPES_WITH_CHART_REFERENCE
):
continue
layer_value = layer.get("value")
+ source_scope[str(layer_value)] = self._annotation_source_scope(layer)
+ if source_scope:
+ context["source_scope"] = source_scope
+
+ return context
+
+ def _annotation_source_scope(self, layer: dict[str, Any]) -> dict[str, Any]:
+ """
+ Access and data-identity cache-key material for one chart-backed
+ annotation layer.
+
+ ``access`` keeps a user denied the referenced chart's datasource from
+ reading an authorized user's cached payload. When the chart has a
+ saved query context, this runs the *same* authorization path
+ :meth:`get_viz_annotation_data` executes
+ (``QueryContext.raise_for_access``) rather than the coarser
+ :meth:`SecurityManager.can_access_datasource` β a requester whose
+ access comes from a dashboard/viewer-promiscuous-mode bypass (which
+ depends on the chart's own saved ``form_data``, e.g. its
+ ``slice_id``/``dashboardId``) would otherwise still fail that coarser,
+ context-free check and collapse onto the same denied scope as a
+ genuinely unauthorized requester, letting the latter read the
+ former's cached payload. ``data_key`` is the annotation chart's own
+ query cache key(s) β derived from the same, override-applied query
+ objects actually executed (see :meth:`_apply_annotation_overrides`),
+ so it captures the datasource version, RLS clauses, and any per-user
+ Jinja/virtual-dataset RLS material exactly as the live fetch would,
+ including material an override only introduces at a finer grain.
+ Reusing this logic avoids re-deriving it and automatically inherits
+ any future correctness fixes made there.
+ """
+ layer_value = layer.get("value")
+ datasource = None
+ try:
chart = (
ChartDAO.find_by_id(layer_value) if layer_value is not None else None
)
- annotation_datasource = chart.datasource if chart else None
- source_rls[str(layer.get("value"))] = (
- security_manager.get_rls_cache_key(annotation_datasource)
- if annotation_datasource
- else None
+ # resolved_datasource, not datasource: the latter is pinned to
+ # table-backed datasources and resolves to None for a
+ # semantic-view-backed chart, which would otherwise collapse
+ # every requester onto the same {access: None, data_key: None}
+ # scope below regardless of their actual access.
+ datasource = chart.resolved_datasource if chart else None
+ if chart is None or datasource is None:
+ return {"access": None, "data_key": None}
+
+ annotation_query_context = chart.get_query_context()
+ if annotation_query_context is not None:
+ self._apply_annotation_overrides(annotation_query_context, layer)
+ try:
+ annotation_query_context.raise_for_access()
+ access: Any = True
+ except SupersetSecurityException:
+ access = False
+ data_key: Any = [
+ annotation_query_context.query_cache_key(query_object)
Review Comment:
You are right, I had the host chart in my head. The key is now derived from the query after result-type preparation (`prepare_query_object`, shared with the result handler), with a test covering a samples source chart whose metrics get cleared.
##########
superset/common/query_context_processor.py:
##########
@@ -474,73 +503,226 @@ def query_cache_key(self, query_obj: QueryObject, **kwargs: Any) -> str | None:
)
return cache_key
- def _annotation_cache_context(self, query_obj: QueryObject) -> dict[str, Any]:
+ def annotation_cache_key(self, query_obj: QueryObject) -> str | None:
"""
- Cache-key material binding cached annotation data to its security
- context.
-
- Annotation payloads are fetched per requesting user and stored on the
- same cache entry as the dataframe, so the key also binds the requesting
- user and, for chart-backed layers, the RLS clauses of the referenced
- chart's datasource.
+ Cache key for this query's annotation-layer payload, or ``None`` when
+ the query has no annotation layers or when some layer's access scope
+ could not be resolved (the payload is then never cached, see
+ :meth:`_annotation_source_scope`).
+
+ Annotation payloads are fetched under the requesting user's access
+ scope, which is a stricter security requirement than the dataframe
+ itself has. Keying them separately from :meth:`query_cache_key` keeps
+ that scoping from forcing every distinct viewer of an annotated chart
+ onto their own full copy of the (potentially much larger) shared
+ dataframe: users with the same access scope share this key too.
"""
- source_rls: dict[str, list[str] | None] = {}
- source_versions: dict[str, tuple[str, str]] = {}
- metadata_versions: dict[str, str] = {}
+ if not query_obj or not query_obj.annotation_layers:
+ return None
+ if (annotation_context := self._annotation_cache_context(query_obj)) is None:
+ return None
+ return self.query_cache_key(query_obj, annotation_context=annotation_context)
+
+ def _annotation_cache_context(
+ self, query_obj: QueryObject
+ ) -> dict[str, Any] | None:
+ """
+ Cache-key material binding annotation data to its security *scope* so
+ users with the same access share a cache entry and users with a
+ different scope β or no access β never read each other's data.
+
+ * NATIVE layers: the ``can_read`` permission on ``Annotation``, the
+ only user-dependent dimension of these global records.
+ * Chart-backed (``line``/``table``) layers: see
+ :meth:`_annotation_source_scope`, recorded per *layer* (not per
+ referenced chart) since two layers can reference the same chart
+ with different overrides, and so different scopes.
+
+ Returns ``None`` when any layer's scope cannot be resolved, meaning
+ the payload must not be cached at all.
+ """
+ context: dict[str, Any] = {}
+
+ if any(
+ layer.get("sourceType") == "NATIVE" for layer in query_obj.annotation_layers
+ ):
+ context["annotation_read"] = security_manager.can_access(
+ "can_read", "Annotation"
+ )
+
+ source_scope: list[dict[str, Any]] = []
for layer in query_obj.annotation_layers:
if (
layer.get("sourceType")
not in ANNOTATION_SOURCE_TYPES_WITH_CHART_REFERENCE
):
continue
- layer_value = layer.get("value")
+ if (scope := self._annotation_source_scope(layer)) is None:
+ return None
+ source_scope.append({"value": str(layer.get("value")), **scope})
+ if source_scope:
+ context["source_scope"] = source_scope
+
+ return context
+
+ def _annotation_source_scope(self, layer: dict[str, Any]) -> dict[str, Any] | None:
+ """
+ Access and data-identity cache-key material for one chart-backed
+ annotation layer, or ``None`` when it cannot be resolved.
+
+ ``None`` means "do not cache": a missing chart, a chart with neither
+ a saved query context nor a resolvable datasource, or any error while
+ deriving the scope would otherwise collapse an authorized and a
+ denied requester onto one shared identity, letting the latter read
+ the former's warmed payload. The caller fetches live instead.
+
+ ``access`` keeps a user denied the referenced chart's datasource from
+ reading an authorized user's cached payload. When the chart has a
+ saved query context, this runs the *same* authorization path
+ :meth:`get_viz_annotation_data` executes
+ (``QueryContext.raise_for_access``) rather than the coarser
+ :meth:`SecurityManager.can_access_datasource` β a requester whose
+ access comes from a dashboard/viewer-promiscuous-mode bypass (which
+ depends on the chart's own saved ``form_data``, e.g. its
+ ``slice_id``/``dashboardId``) would otherwise still fail that coarser,
+ context-free check and collapse onto the same denied scope as a
+ genuinely unauthorized requester, letting the latter read the
+ former's cached payload. ``data_key`` is the annotation chart's own
+ query cache key(s) β derived from the same, override-applied query
+ objects actually executed (see :meth:`_apply_annotation_overrides`),
+ so it captures the datasource version, RLS clauses, and any per-user
+ Jinja/virtual-dataset RLS material exactly as the live fetch would,
+ including material an override only introduces at a finer grain.
+ Reusing this logic avoids re-deriving it and automatically inherits
+ any future correctness fixes made there.
+ """
+ layer_value = layer.get("value")
+ try:
chart = (
ChartDAO.find_by_id(layer_value) if layer_value is not None else None
)
- annotation_datasource: Datasource | None = (
- chart.resolved_datasource if chart else None
- )
- if chart and isinstance(chart.query_context, str):
+ if chart is None:
+ return None
+
+ # The saved query context is what the live fetch executes and
+ # authorizes against, so it is consulted first, independent of
+ # the chart's own datasource pointer: a chart can have its
+ # ``datasource_id`` cleared while its saved query context still
+ # targets a valid dataset, and the live fetch would still work.
+ annotation_query_context = chart.get_query_context()
+ if annotation_query_context is not None:
+ self._apply_annotation_overrides(annotation_query_context, layer)
try:
- source: dict[str, Any] = json.loads(chart.query_context)[
- "datasource"
- ]
- annotation_datasource = DatasourceDAO.get_datasource(
- source["type"], source["id"]
- )
- except (
- ValueError,
- TypeError,
- KeyError,
- DatasourceNotFound,
- DatasourceTypeNotSupportedError,
- DatasourceValueIsIncorrect,
- ):
- annotation_datasource = None
+ annotation_query_context.raise_for_access()
+ access: Any = True
+ except SupersetSecurityException:
+ access = False
+ data_key: Any = [
+ annotation_query_context.query_cache_key(query_object)
Review Comment:
Fair point on the `-1` case. A referenced chart (or its datasource) with caching disabled now yields no scope at all, so its annotation payload is fetched live and never cached under the host chart. Tests cover both the saved-query-context path and the datasource fallback.
##########
superset/charts/data/api.py:
##########
@@ -448,6 +448,15 @@ def _run_async(
return self._send_chart_response(result)
except ChartDataCacheLoadError:
pass
+ except (
Review Comment:
Added `test_run_async_answers_400_when_cached_read_back_fails`: `execute(force_cached=True)` raises `ChartDataQueryFailedError`, and it asserts a 400 with no task submitted.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] fix(charts): decouple annotation cache from the shared dataframe cache [superset]
Posted by "rusackas (via GitHub)" <gi...@apache.org>.
rusackas commented on code in PR #44406:
URL: https://github.com/apache/superset/pull/44406#discussion_r4297988255
##########
tests/unit_tests/common/test_query_context_processor.py:
##########
@@ -3115,11 +3788,17 @@ def payload(context: QueryContext, **kwargs: Any) -> dict[str, Any]:
"superset.common.query_context_processor.security_manager.get_rls_cache_key",
return_value=[],
),
+ patch.object(QueryContext, "raise_for_access"),
+ patch.object(QueryContext, "query_cache_key", return_value="data-key"),
+ patch(
+ "superset.common.query_context_processor.security_manager"
+ ".can_access_datasource",
+ return_value=True,
+ ),
patch(
"superset.common.query_context_processor.ChartDAO.find_by_id",
return_value=chart,
),
- patch("superset.common.query_context_processor.get_user_id", return_value=42),
):
context: dict[str, Any] = processor._annotation_cache_context(query)
Review Comment:
This was fixed in the mypy follow-up commit (3b1a4d59), which narrows the optional before indexing.
##########
tests/unit_tests/common/test_query_context_processor.py:
##########
@@ -2639,6 +3215,101 @@ def test_mark_force_executed_noop_without_nonce(processor, mock_query_context):
cache_manager.data_cache.set.assert_not_called()
+# =============================================================================
+# Annotation-data cache decoupled from the dataframe cache
+# =============================================================================
+
+
+def test_get_annotation_data_cached_reads_from_cache(
+ processor: QueryContextProcessor,
+) -> None:
+ """A hit on the annotation-specific key skips recomputation entirely."""
+ with patch(
+ "superset.common.query_context_processor.cache_manager"
+ ) as cache_manager:
+ cache_manager.data_cache.get.return_value = {"annotation_data": {"a": 1}}
+ with patch.object(processor, "get_annotation_data") as mock_get:
+ result = processor._get_annotation_data_cached(
+ query_obj=MagicMock(),
+ cache_key="ak",
+ force_query=False,
+ force_cached=False,
+ timeout=60,
+ datasource_uid="ds",
+ )
+ assert result == {"a": 1}
+ mock_get.assert_not_called()
+
+
+def test_get_annotation_data_cached_computes_and_caches_on_miss(
+ processor: QueryContextProcessor,
+) -> None:
+ """A miss computes the annotation data and stores it under the scoped key."""
+ with (
+ patch("superset.common.query_context_processor.cache_manager") as cache_manager,
+ patch("superset.common.query_context_processor.set_and_log_cache") as mock_set,
+ ):
+ cache_manager.data_cache.get.return_value = None
+ with patch.object(
+ processor, "get_annotation_data", return_value={"a": 1}
+ ) as mock_get:
+ result = processor._get_annotation_data_cached(
+ query_obj=MagicMock(),
+ cache_key="ak",
+ force_query=False,
+ force_cached=False,
+ timeout=60,
+ datasource_uid="ds",
+ )
+ assert result == {"a": 1}
+ mock_get.assert_called_once()
+ mock_set.assert_called_once_with(
+ cache_manager.data_cache, "ak", {"annotation_data": {"a": 1}}, 60, "ds"
+ )
+
+
+def test_get_annotation_data_cached_force_cached_raises_on_miss(
+ processor: QueryContextProcessor,
+) -> None:
+ """``force_cached`` must never fall through to a live compute -- the same
+ contract ``QueryCacheManager.get`` enforces for the dataframe cache."""
+ with patch(
+ "superset.common.query_context_processor.cache_manager"
+ ) as cache_manager:
+ cache_manager.data_cache.get.return_value = None
+ with pytest.raises(CacheLoadError):
+ processor._get_annotation_data_cached(
+ query_obj=MagicMock(),
+ cache_key="ak",
+ force_query=False,
+ force_cached=True,
+ timeout=60,
+ datasource_uid="ds",
+ )
+
+
+def test_get_annotation_data_cached_force_query_bypasses_cache_read(processor):
+ """A forced refresh recomputes rather than serving a stale cached value."""
+ with (
+ patch("superset.common.query_context_processor.cache_manager") as cache_manager,
+ patch("superset.common.query_context_processor.set_and_log_cache"),
Review Comment:
Added a parametrized test where `set_and_log_cache` returns True and False, asserting the marker call gets that value under the annotation key.
##########
superset/common/query_context_processor.py:
##########
@@ -474,73 +503,226 @@ def query_cache_key(self, query_obj: QueryObject, **kwargs: Any) -> str | None:
)
return cache_key
- def _annotation_cache_context(self, query_obj: QueryObject) -> dict[str, Any]:
+ def annotation_cache_key(self, query_obj: QueryObject) -> str | None:
"""
- Cache-key material binding cached annotation data to its security
- context.
-
- Annotation payloads are fetched per requesting user and stored on the
- same cache entry as the dataframe, so the key also binds the requesting
- user and, for chart-backed layers, the RLS clauses of the referenced
- chart's datasource.
+ Cache key for this query's annotation-layer payload, or ``None`` when
+ the query has no annotation layers or when some layer's access scope
+ could not be resolved (the payload is then never cached, see
+ :meth:`_annotation_source_scope`).
+
+ Annotation payloads are fetched under the requesting user's access
+ scope, which is a stricter security requirement than the dataframe
+ itself has. Keying them separately from :meth:`query_cache_key` keeps
+ that scoping from forcing every distinct viewer of an annotated chart
+ onto their own full copy of the (potentially much larger) shared
+ dataframe: users with the same access scope share this key too.
"""
- source_rls: dict[str, list[str] | None] = {}
- source_versions: dict[str, tuple[str, str]] = {}
- metadata_versions: dict[str, str] = {}
+ if not query_obj or not query_obj.annotation_layers:
+ return None
+ if (annotation_context := self._annotation_cache_context(query_obj)) is None:
+ return None
+ return self.query_cache_key(query_obj, annotation_context=annotation_context)
+
+ def _annotation_cache_context(
+ self, query_obj: QueryObject
+ ) -> dict[str, Any] | None:
+ """
+ Cache-key material binding annotation data to its security *scope* so
+ users with the same access share a cache entry and users with a
+ different scope β or no access β never read each other's data.
+
+ * NATIVE layers: the ``can_read`` permission on ``Annotation``, the
+ only user-dependent dimension of these global records.
+ * Chart-backed (``line``/``table``) layers: see
+ :meth:`_annotation_source_scope`, recorded per *layer* (not per
+ referenced chart) since two layers can reference the same chart
+ with different overrides, and so different scopes.
+
+ Returns ``None`` when any layer's scope cannot be resolved, meaning
+ the payload must not be cached at all.
+ """
+ context: dict[str, Any] = {}
+
+ if any(
+ layer.get("sourceType") == "NATIVE" for layer in query_obj.annotation_layers
+ ):
+ context["annotation_read"] = security_manager.can_access(
+ "can_read", "Annotation"
+ )
+
+ source_scope: list[dict[str, Any]] = []
for layer in query_obj.annotation_layers:
if (
layer.get("sourceType")
not in ANNOTATION_SOURCE_TYPES_WITH_CHART_REFERENCE
):
continue
- layer_value = layer.get("value")
+ if (scope := self._annotation_source_scope(layer)) is None:
+ return None
+ source_scope.append({"value": str(layer.get("value")), **scope})
+ if source_scope:
+ context["source_scope"] = source_scope
+
+ return context
+
+ def _annotation_source_scope(self, layer: dict[str, Any]) -> dict[str, Any] | None:
+ """
+ Access and data-identity cache-key material for one chart-backed
+ annotation layer, or ``None`` when it cannot be resolved.
+
+ ``None`` means "do not cache": a missing chart, a chart with neither
+ a saved query context nor a resolvable datasource, or any error while
+ deriving the scope would otherwise collapse an authorized and a
+ denied requester onto one shared identity, letting the latter read
+ the former's warmed payload. The caller fetches live instead.
+
+ ``access`` keeps a user denied the referenced chart's datasource from
+ reading an authorized user's cached payload. When the chart has a
+ saved query context, this runs the *same* authorization path
+ :meth:`get_viz_annotation_data` executes
+ (``QueryContext.raise_for_access``) rather than the coarser
+ :meth:`SecurityManager.can_access_datasource` β a requester whose
+ access comes from a dashboard/viewer-promiscuous-mode bypass (which
+ depends on the chart's own saved ``form_data``, e.g. its
+ ``slice_id``/``dashboardId``) would otherwise still fail that coarser,
+ context-free check and collapse onto the same denied scope as a
+ genuinely unauthorized requester, letting the latter read the
+ former's cached payload. ``data_key`` is the annotation chart's own
+ query cache key(s) β derived from the same, override-applied query
+ objects actually executed (see :meth:`_apply_annotation_overrides`),
+ so it captures the datasource version, RLS clauses, and any per-user
+ Jinja/virtual-dataset RLS material exactly as the live fetch would,
+ including material an override only introduces at a finer grain.
+ Reusing this logic avoids re-deriving it and automatically inherits
+ any future correctness fixes made there.
+ """
+ layer_value = layer.get("value")
+ try:
chart = (
ChartDAO.find_by_id(layer_value) if layer_value is not None else None
)
- annotation_datasource: Datasource | None = (
- chart.resolved_datasource if chart else None
- )
- if chart and isinstance(chart.query_context, str):
+ if chart is None:
+ return None
+
+ # The saved query context is what the live fetch executes and
+ # authorizes against, so it is consulted first, independent of
+ # the chart's own datasource pointer: a chart can have its
+ # ``datasource_id`` cleared while its saved query context still
+ # targets a valid dataset, and the live fetch would still work.
+ annotation_query_context = chart.get_query_context()
+ if annotation_query_context is not None:
+ self._apply_annotation_overrides(annotation_query_context, layer)
try:
- source: dict[str, Any] = json.loads(chart.query_context)[
- "datasource"
- ]
- annotation_datasource = DatasourceDAO.get_datasource(
- source["type"], source["id"]
- )
- except (
- ValueError,
- TypeError,
- KeyError,
- DatasourceNotFound,
- DatasourceTypeNotSupportedError,
- DatasourceValueIsIncorrect,
- ):
- annotation_datasource = None
+ annotation_query_context.raise_for_access()
+ access: Any = True
+ except SupersetSecurityException:
+ access = False
+ data_key: Any = [
Review Comment:
Agreed. A denied scope now returns `{"access": False, "data_key": None}` without calling `query_cache_key`, so no Jinja gets rendered for a dataset the requester can't see. Test added.
##########
tests/unit_tests/common/test_query_context_processor.py:
##########
@@ -2639,6 +3215,101 @@ def test_mark_force_executed_noop_without_nonce(processor, mock_query_context):
cache_manager.data_cache.set.assert_not_called()
+# =============================================================================
+# Annotation-data cache decoupled from the dataframe cache
+# =============================================================================
+
+
+def test_get_annotation_data_cached_reads_from_cache(
+ processor: QueryContextProcessor,
+) -> None:
+ """A hit on the annotation-specific key skips recomputation entirely."""
+ with patch(
+ "superset.common.query_context_processor.cache_manager"
+ ) as cache_manager:
+ cache_manager.data_cache.get.return_value = {"annotation_data": {"a": 1}}
+ with patch.object(processor, "get_annotation_data") as mock_get:
+ result = processor._get_annotation_data_cached(
+ query_obj=MagicMock(),
+ cache_key="ak",
+ force_query=False,
+ force_cached=False,
+ timeout=60,
+ datasource_uid="ds",
+ )
+ assert result == {"a": 1}
+ mock_get.assert_not_called()
+
+
+def test_get_annotation_data_cached_computes_and_caches_on_miss(
+ processor: QueryContextProcessor,
+) -> None:
+ """A miss computes the annotation data and stores it under the scoped key."""
+ with (
+ patch("superset.common.query_context_processor.cache_manager") as cache_manager,
+ patch("superset.common.query_context_processor.set_and_log_cache") as mock_set,
+ ):
+ cache_manager.data_cache.get.return_value = None
+ with patch.object(
+ processor, "get_annotation_data", return_value={"a": 1}
+ ) as mock_get:
+ result = processor._get_annotation_data_cached(
+ query_obj=MagicMock(),
+ cache_key="ak",
+ force_query=False,
+ force_cached=False,
+ timeout=60,
+ datasource_uid="ds",
+ )
+ assert result == {"a": 1}
+ mock_get.assert_called_once()
+ mock_set.assert_called_once_with(
+ cache_manager.data_cache, "ak", {"annotation_data": {"a": 1}}, 60, "ds"
+ )
+
+
+def test_get_annotation_data_cached_force_cached_raises_on_miss(
Review Comment:
Added a case where `data_cache.get` raises: live annotations come back without `force_cached`, and `CacheLoadError` is raised (no live fetch) with it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] fix(charts): decouple annotation cache from the shared dataframe cache [superset]
Posted by "bito-code-review[bot] (via GitHub)" <gi...@apache.org>.
bito-code-review[bot] commented on code in PR #44406:
URL: https://github.com/apache/superset/pull/44406#discussion_r4045403578
##########
superset/common/query_context_processor.py:
##########
@@ -445,31 +464,144 @@ def query_cache_key(self, query_obj: QueryObject, **kwargs: Any) -> str | None:
)
return cache_key
- def _annotation_cache_context(self, query_obj: QueryObject) -> dict[str, Any]:
+ def annotation_cache_key(self, query_obj: QueryObject) -> str | None:
+ """
+ Cache key for this query's annotation-layer payload, or ``None`` when
+ the query has no annotation layers.
+
+ Annotation payloads are fetched under the requesting user's access
+ scope, which is a stricter security requirement than the dataframe
+ itself has. Keying them separately from :meth:`query_cache_key` keeps
+ that scoping from forcing every distinct viewer of an annotated chart
+ onto their own full copy of the (potentially much larger) shared
+ dataframe: users with the same access scope share this key too.
"""
- Cache-key material binding cached annotation data to its security
- context.
+ if not query_obj or not query_obj.annotation_layers:
+ return None
+ return self.query_cache_key(
+ query_obj, annotation_context=self._annotation_cache_context(query_obj)
+ )
- Annotation payloads are fetched per requesting user and stored on the
- same cache entry as the dataframe, so the key also binds the requesting
- user and, for chart-backed layers, the RLS clauses of the referenced
- chart's datasource.
+ def _annotation_cache_context(self, query_obj: QueryObject) -> dict[str, Any]:
+ """
+ Cache-key material binding annotation data to its security *scope* so
+ users with the same access share a cache entry and users with a
+ different scope β or no access β never read each other's data.
+
+ * NATIVE layers: the ``can_read`` permission on ``Annotation``, the
+ only user-dependent dimension of these global records.
+ * Chart-backed (``line``/``table``) layers: see
+ :meth:`_annotation_source_scope`.
"""
- source_rls: dict[str, list[str] | None] = {}
+ context: dict[str, Any] = {}
+
+ if any(
+ layer.get("sourceType") == "NATIVE" for layer in query_obj.annotation_layers
+ ):
+ context["annotation_read"] = security_manager.can_access(
+ "can_read", "Annotation"
+ )
+
+ source_scope: dict[str, Any] = {}
for layer in query_obj.annotation_layers:
if layer.get("sourceType") not in ("line", "table"):
continue
layer_value = layer.get("value")
- chart = (
- ChartDAO.find_by_id(layer_value) if layer_value is not None else None
+ source_scope[str(layer_value)] = self._annotation_source_scope(layer_value)
+ if source_scope:
+ context["source_scope"] = source_scope
+
+ return context
+
+ def _annotation_source_scope(self, layer_value: Any) -> dict[str, Any]:
+ """
+ Access and data-identity cache-key material for one chart-backed
+ annotation layer.
+
+ ``access`` keeps a user denied the referenced chart's datasource from
+ reading an authorized user's cached payload. ``data_key`` is the
+ annotation chart's own query cache key(s), which already capture the
+ datasource version, RLS clauses, and any per-user Jinja/virtual-dataset
+ RLS material β reusing it here avoids re-deriving that logic and
+ automatically inherits any future correctness fixes made there.
+ """
+ chart = ChartDAO.find_by_id(layer_value) if layer_value is not None else None
+ datasource = chart.datasource if chart else None
+ if chart is None or datasource is None:
+ return {"access": None, "data_key": None}
+
+ try:
+ access = security_manager.can_access_datasource(datasource)
+ # Fall back to the RLS-clause identity when the chart has no saved
+ # query context to key on.
+ annotation_query_context = chart.get_query_context()
+ data_key: Any = (
+ [
+ annotation_query_context.query_cache_key(query_object)
+ for query_object in annotation_query_context.queries
+ ]
+ if annotation_query_context is not None
+ else security_manager.get_rls_cache_key(datasource)
+ )
+ except SupersetException:
Review Comment:
<!-- Bito Reply -->
The suggestion to widen the exception to a bare `except` is appropriate here. It ensures that any unexpected errors during the derivation of the cache keyβsuch as those from SQLAlchemy or Jinjaβare caught, allowing the system to fall back to the default behavior instead of causing a 500 error for the entire request.
**superset/common/query_context_processor.py**
```
except Exception:
return {"access": None, "data_key": None}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] fix(charts): decouple annotation cache from the shared dataframe cache [superset]
Posted by "bito-code-review[bot] (via GitHub)" <gi...@apache.org>.
bito-code-review[bot] commented on PR #44406:
URL: https://github.com/apache/superset/pull/44406#issuecomment-5724176950
AI Code Review is in progress (usually takes 3 to 15 minutes unless it's a very large PR).
<details>
<summary>
Bito Usage Guide
</summary>
**Commands**
Type the following command in the pull request comment and save the comment.
- `/review` - Manually triggers an incremental AI Review.
- `/review full` - Manually triggers a full AI Review.
- `/pause` - Pauses automatic reviews on this pull request.
- `/resume` - Resumes automatic reviews.
- `/resolve` - Marks all Bito-posted review comments as resolved.
- `/abort` - Cancels all in-progress reviews.
Refer to the <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/available-commands" target="_blank" rel="noopener noreferrer">documentation</a> for additional commands.
**Configuration**
This repository uses `Superset` You can customize the agent settings <a href="https://alpha.bito.ai/home/ai-agents/code-review-agent" target="_blank" rel="noopener noreferrer">here</a> or contact your Bito workspace admin at evan@preset.io.
**Documentation & Help**
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/getting-started/install-run-using-bito-cloud/create-or-customize-an-agent-instance" target="_blank" rel="noopener noreferrer">Customize agent settings</a>
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/implementing-custom-code-review-rules" target="_blank" rel="noopener noreferrer">Review rules</a>
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent" target="_blank" rel="noopener noreferrer">General documentation</a>
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/faqs" target="_blank" rel="noopener noreferrer">FAQ</a>
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] fix(charts): decouple annotation cache from the shared dataframe cache [superset]
Posted by "rusackas (via GitHub)" <gi...@apache.org>.
rusackas commented on code in PR #44406:
URL: https://github.com/apache/superset/pull/44406#discussion_r4172344817
##########
tests/unit_tests/common/test_query_context_processor.py:
##########
@@ -2630,6 +3041,94 @@ def test_mark_force_executed_noop_without_nonce(processor, mock_query_context):
cache_manager.data_cache.set.assert_not_called()
+# =============================================================================
+# Annotation-data cache decoupled from the dataframe cache
+# =============================================================================
+
+
+def test_get_annotation_data_cached_reads_from_cache(processor):
+ """A hit on the annotation-specific key skips recomputation entirely."""
+ with patch(
+ "superset.common.query_context_processor.cache_manager"
+ ) as cache_manager:
+ cache_manager.data_cache.get.return_value = {"annotation_data": {"a": 1}}
+ with patch.object(processor, "get_annotation_data") as mock_get:
+ result = processor._get_annotation_data_cached(
+ query_obj=MagicMock(),
+ cache_key="ak",
+ force_query=False,
+ force_cached=False,
+ timeout=60,
+ datasource_uid="ds",
+ )
+ assert result == {"a": 1}
+ mock_get.assert_not_called()
+
+
+def test_get_annotation_data_cached_computes_and_caches_on_miss(processor):
Review Comment:
Fixed in the latest push.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] fix(charts): decouple annotation cache from the shared dataframe cache [superset]
Posted by "bito-code-review[bot] (via GitHub)" <gi...@apache.org>.
bito-code-review[bot] commented on PR #44406:
URL: https://github.com/apache/superset/pull/44406#issuecomment-5968822502
AI Code Review is in progress (usually takes 3 to 15 minutes unless it's a very large PR).
<details>
<summary>
Bito Usage Guide
</summary>
**Commands**
Type the following command in the pull request comment and save the comment.
- `/review` - Manually triggers an incremental AI Review.
- `/review full` - Manually triggers a full AI Review.
- `/pause` - Pauses automatic reviews on this pull request.
- `/resume` - Resumes automatic reviews.
- `/resolve` - Marks all Bito-posted review comments as resolved.
- `/abort` - Cancels all in-progress reviews.
Refer to the <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/available-commands" target="_blank" rel="noopener noreferrer">documentation</a> for additional commands.
**Configuration**
This repository uses `Superset` You can customize the agent settings <a href="https://alpha.bito.ai/home/ai-agents/code-review-agent" target="_blank" rel="noopener noreferrer">here</a> or contact your Bito workspace admin at evan@preset.io.
**Documentation & Help**
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/getting-started/install-run-using-bito-cloud/create-or-customize-an-agent-instance" target="_blank" rel="noopener noreferrer">Customize agent settings</a>
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/implementing-custom-code-review-rules" target="_blank" rel="noopener noreferrer">Review rules</a>
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent" target="_blank" rel="noopener noreferrer">General documentation</a>
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/faqs" target="_blank" rel="noopener noreferrer">FAQ</a>
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] fix(charts): decouple annotation cache from the shared dataframe cache [superset]
Posted by "sadpandajoe (via GitHub)" <gi...@apache.org>.
sadpandajoe commented on code in PR #44406:
URL: https://github.com/apache/superset/pull/44406#discussion_r4217596322
##########
tests/unit_tests/common/test_query_context_processor.py:
##########
@@ -117,25 +122,350 @@ def processor(mock_query_context):
return processor
-def test_query_cache_key_binds_annotation_data_to_requesting_user(processor):
- """The cache key for annotated queries must differ per requesting user."""
+def test_annotation_cache_key_binds_native_annotation_read_scope(processor) -> None:
+ """The annotation cache key for NATIVE layers must differ when the
+ requester's ``can_read`` (Annotation) access differs -- not who they are."""
query_obj = MagicMock()
query_obj.annotation_layers = [{"sourceType": "NATIVE", "name": "a", "value": 1}]
- with (
- patch(
- "superset.common.query_context_processor.get_user_id",
- side_effect=[1, 2],
- ),
- patch("superset.common.query_context_processor.security_manager"),
- ):
- processor.query_cache_key(query_obj)
- processor.query_cache_key(query_obj)
+ # ``security_manager`` autodetects as an async spec under a bare
+ # ``patch()`` (its real object trips ``unittest.mock``'s coroutine
+ # inference), which would silently turn every attribute access into an
+ # ``AsyncMock`` returning a fresh unawaited coroutine per call -- always
+ # unequal to itself and never equal to a configured return value. Forcing
+ # ``new_callable=MagicMock`` keeps these synchronous, as the real object
+ # is.
+ with patch(
+ "superset.common.query_context_processor.security_manager",
+ new_callable=MagicMock,
+ ) as security_manager:
+ security_manager.can_access.side_effect = [True, False]
+ processor.annotation_cache_key(query_obj)
+ processor.annotation_cache_key(query_obj)
contexts = [
call.kwargs["annotation_context"] for call in query_obj.cache_key.call_args_list
]
assert contexts[0] != contexts[1]
+def test_annotation_cache_key_shares_across_same_access_scope() -> None:
+ """Two distinct requesters (separate processor/query-object instances,
+ standing in for two different requests) with identical access scope must
+ produce identical annotation-context material. Reusing a single
+ processor/query_obj across both calls (as this test previously did)
+ would pass trivially regardless of whether the key is scope-based or
+ identity-based, since nothing about "who's asking" would ever vary."""
+ layer = {"sourceType": "NATIVE", "name": "a", "value": 1}
+ processor_a = QueryContextProcessor(MagicMock())
+ processor_b = QueryContextProcessor(MagicMock())
+ query_obj_a = MagicMock(annotation_layers=[layer])
+ query_obj_b = MagicMock(annotation_layers=[layer])
+
+ with patch(
+ "superset.common.query_context_processor.security_manager",
+ new_callable=MagicMock,
+ ) as security_manager:
+ security_manager.can_access.return_value = True
+ context_a = processor_a._annotation_cache_context(query_obj_a)
+ context_b = processor_b._annotation_cache_context(query_obj_b)
+
+ assert context_a == context_b
+
+
+def test_query_cache_key_does_not_bind_annotation_scope(processor) -> None:
+ """The dataframe cache key must stay shared across viewers of the same
+ chart, even when the query has annotation layers β only the separate
+ annotation cache key (see above) carries access-scope material."""
+ query_obj = MagicMock()
+ query_obj.annotation_layers = [{"sourceType": "NATIVE", "name": "a", "value": 1}]
+ with patch(
+ "superset.common.query_context_processor.security_manager",
+ new_callable=MagicMock,
+ ):
+ processor.query_cache_key(query_obj)
+ processor.query_cache_key(query_obj)
+ for call in query_obj.cache_key.call_args_list:
+ assert "annotation_context" not in call.kwargs
+
+
+@pytest.fixture
+def mock_annotation_chart() -> Iterator[MagicMock]:
+ """A found chart, wired as the referenced chart for
+ ``_annotation_source_scope`` tests -- factors out the repeated
+ ``ChartDAO.find_by_id`` patch those tests all need."""
+ chart = MagicMock()
+ with patch(
+ "superset.common.query_context_processor.ChartDAO.find_by_id",
+ return_value=chart,
+ ):
+ yield chart
+
+
+def test_annotation_source_scope_binds_datasource_access(
+ processor: QueryContextProcessor, mock_annotation_chart: MagicMock
+) -> None:
+ """A chart-backed annotation layer's scope must differ when the
+ requester's access to the referenced datasource differs."""
+ mock_annotation_chart.get_query_context.return_value = None
+ with patch(
+ "superset.common.query_context_processor.security_manager",
+ new_callable=MagicMock,
+ ) as security_manager:
+ security_manager.can_access_datasource.side_effect = [True, False]
+ security_manager.get_rls_cache_key.return_value = []
+ scope_a = processor._annotation_source_scope({"value": 1})
+ scope_b = processor._annotation_source_scope({"value": 1})
+ assert scope_a is not None
+ assert scope_b is not None
+ assert scope_a != scope_b
+ assert scope_a["access"] is True
+ assert scope_b["access"] is False
+
+
+def test_annotation_source_scope_reuses_referenced_chart_cache_key(
+ processor: QueryContextProcessor, mock_annotation_chart: MagicMock
+) -> None:
+ """When the referenced chart has a saved query context, its own cache
+ key(s) -- covering RLS and per-user Jinja/virtual-dataset material -- are
+ reused rather than re-derived."""
+ mock_query_object: MagicMock = MagicMock()
+ mock_query_context: MagicMock = MagicMock()
+ mock_query_context.queries = [mock_query_object]
+ mock_query_context.query_cache_key.return_value = "referenced-chart-key"
+ mock_annotation_chart.get_query_context.return_value = mock_query_context
+ with patch(
+ "superset.common.query_context_processor.security_manager",
+ new_callable=MagicMock,
+ ) as security_manager:
+ security_manager.can_access_datasource.return_value = True
+ scope = processor._annotation_source_scope({"value": 1})
+ assert scope == {"access": True, "data_key": ["referenced-chart-key"]}
+ mock_query_context.query_cache_key.assert_called_once_with(mock_query_object)
+
+
+def test_annotation_source_scope_uses_live_fetch_authorization(
+ processor: QueryContextProcessor, mock_annotation_chart: MagicMock
+) -> None:
+ """When the referenced chart has a saved query context, ``access`` must
+ come from that context's own ``raise_for_access`` -- the same
+ authorization path the live fetch in ``get_viz_annotation_data`` uses --
+ not the coarser, context-free ``can_access_datasource``. A requester
+ denied by ``can_access_datasource`` but granted via a bypass that depends
+ on the chart's own saved form_data (e.g. a dashboard/viewer-promiscuous
+ bypass) must get a scope distinct from one truly denied, or the latter
+ could read the former's cached payload."""
+ mock_query_object: MagicMock = MagicMock()
+ mock_query_context: MagicMock = MagicMock()
+ mock_query_context.queries = [mock_query_object]
+ mock_query_context.query_cache_key.return_value = "referenced-chart-key"
+ mock_annotation_chart.get_query_context.return_value = mock_query_context
+ with patch(
+ "superset.common.query_context_processor.security_manager",
+ new_callable=MagicMock,
+ ) as security_manager:
+ # Coarse proxy says "denied" for both -- the real authorization
+ # path must be what actually decides access.
+ security_manager.can_access_datasource.return_value = False
+ mock_query_context.raise_for_access.side_effect = [
+ None,
+ SupersetSecurityException(MagicMock()),
+ ]
+ scope_a = processor._annotation_source_scope({"value": 1})
+ scope_b = processor._annotation_source_scope({"value": 1})
+ assert scope_a is not None
+ assert scope_b is not None
+ assert scope_a["access"] is True
+ assert scope_b["access"] is False
+ assert scope_a != scope_b
+ security_manager.can_access_datasource.assert_not_called()
+
+
+def test_annotation_source_scope_applies_overrides_before_keying(
+ processor: QueryContextProcessor, mock_annotation_chart: MagicMock
+) -> None:
+ """A time-grain/time-range override on the annotation layer must be
+ applied to the referenced chart's query objects *before* deriving the
+ cache key, mirroring ``get_viz_annotation_data`` exactly -- otherwise the
+ key can omit per-user Jinja/RLS material an override only introduces at a
+ finer grain."""
+ mock_query_object = MagicMock()
+ mock_query_object.extras = {}
+ mock_query_context = MagicMock()
+ mock_query_context.queries = [mock_query_object]
+ mock_query_context.query_cache_key.return_value = "referenced-chart-key"
+ mock_annotation_chart.get_query_context.return_value = mock_query_context
+ layer = {
+ "value": 1,
+ "overrides": {"time_grain_sqla": "P1D", "time_range": "Last week"},
+ }
+ with patch(
+ "superset.common.query_context_processor.security_manager",
+ new_callable=MagicMock,
+ ):
+ processor._annotation_source_scope(layer)
Review Comment:
`query_cache_key` is mocked to a constant here and the assertions only look at the final query-object state, so moving `_apply_annotation_overrides` after key derivation would still pass. That reintroduces the miss where a daily override pulls in `current_user_id()` Jinja that the saved monthly query lacks, and the second viewer gets the first viewer's rows. Could `query_cache_key` use a `side_effect` that asserts the overridden grain, `time_range` and bounds are already set when it is called?
##########
superset/common/query_context_processor.py:
##########
@@ -332,6 +344,30 @@ def get_df_payload_result(
# nonce reads the freshly-cached result instead of recomputing it.
self._mark_force_executed(query_obj, cache_key, cache.result_persisted)
+ # Annotation data is fetched per requesting user (and, for chart-backed
+ # layers, scoped by the referenced chart datasource's RLS), so it is
+ # resolved and cached under its own entry β independent of whether the
+ # (shareable) dataframe above was a hit or a miss β rather than forcing
+ # every viewer of the same chart onto their own full dataframe copy.
+ annotation_data: dict[str, Any] = {}
+ if (
+ query_obj
+ and query_obj.annotation_layers
+ and cache.status != QueryStatus.FAILED
+ ):
+ try:
+ annotation_data = self._get_annotation_data_cached(
Review Comment:
With `CHART_DATA_INCLUDE_TIMING` on, annotation SQL now runs after `data_acquisition_ns` is captured and before `payload_assembly_start_ns` starts, so it lands in none of the reported phases. A cold chart with a 10 ms dataframe and a 5 s annotation query reports roughly 10 ms of acquisition, and a dataframe cache hit reports `data_acquisition_ms: null` even though annotation SQL ran. The configuring-superset docs say `data_acquisition_ms` "includes database work and annotation dependencies", and version 1 can't change semantics. Should the annotation fetch be timed into `data_acquisition_ns` (summed with the dataframe time when both run)?
##########
tests/unit_tests/common/test_query_context_processor.py:
##########
@@ -2238,6 +2568,252 @@ def test_get_df_payload_no_warning_when_not_memory_limited() -> None:
assert result["warning"] is None
+def test_get_df_payload_result_decouples_annotation_cache_from_dataframe_cache() -> (
+ None
+):
+ """
+ The dataframe cache entry must stay shareable across viewers, and
+ annotation-layer data must be resolved through its own (per-user) cache
+ path -- not stored on the dataframe's cache entry -- so that two viewers
+ of the same annotated chart share one dataframe cache hit while each
+ still gets their own annotation-security-scoped payload.
+ """
+ from superset.common.query_object import QueryObject
+
+ mock_query_context = MagicMock()
+ mock_query_context.force = False
+ mock_datasource = MagicMock()
+ mock_datasource.column_names = ["col1"]
+
+ processor = QueryContextProcessor(mock_query_context)
+ processor._qc_datasource = mock_datasource
+
+ query_obj = QueryObject(
+ datasource=mock_datasource,
+ columns=["col1"],
+ annotation_layers=[
+ {
+ "annotationType": "EVENT",
+ "sourceType": "NATIVE",
+ "name": "a",
+ "value": 1,
+ }
+ ],
+ )
+
+ class MockCache:
+ def __init__(self):
+ self.is_loaded = True
+ self.applied_filter_columns = ["col1"]
+ self.df = pd.DataFrame({"col1": [1, 2, 3]})
+ self.query = ""
+ self.status = "success"
+ self.cache_dttm = "2024-01-01T00:00:00"
+ self.queried_dttm = "2024-01-01T00:00:00"
+ self.stacktrace = None
+ self.error_message = None
+ self.is_cached = True
+ self.sql_rowcount = 0
+ self.cache_value = None
+ self.applied_template_filters = []
+ self.rejected_filter_columns = []
+ self.annotation_data = {"stale": "should not be used"}
+ self.bq_memory_limited = False
+ self.bq_memory_limited_row_count = 0
+ self.result_persisted = False
+ self.set_query_result = MagicMock()
+
+ mock_cache = MockCache()
+
+ with (
+ patch(
+ "superset.common.query_context_processor.QueryCacheManager"
+ ) as mock_cache_manager,
+ patch.object(query_obj, "validate", return_value=None),
+ patch.object(processor, "query_cache_key", return_value="df-key"),
+ patch.object(processor, "annotation_cache_key", return_value="ann-key"),
+ patch.object(
+ processor, "_get_annotation_data_cached", return_value={"a": [1, 2]}
+ ) as mock_get_annotation,
+ patch.object(processor, "get_cache_timeout", return_value=3600),
+ ):
+ mock_cache_manager.get.return_value = mock_cache
+ result = processor.get_df_payload(query_obj, force_cached=False)
+
+ # The dataframe cache is a hit, so the (expensive) query is never re-run,
+ # and its cache entry is never rewritten.
+ mock_cache.set_query_result.assert_not_called()
+
+ # Annotation data is resolved through the separate, per-user cache path,
+ # keyed by the dedicated annotation cache key -- not the dataframe's key.
+ mock_get_annotation.assert_called_once()
+ _, kwargs = mock_get_annotation.call_args
+ assert kwargs["cache_key"] == "ann-key"
Review Comment:
This only asserts `cache_key` on the `_get_annotation_data_cached` call. The helper-level `force_cached` test passes the flag directly and the payload test replaces the helper, so dropping `force_cached=force_cached` at the call site in `get_df_payload_result` would still pass. Async cache preflight with `force_cached=True` and a missing annotation entry would then run the annotation SQL instead of raising `CacheLoadError`. Should this also assert `kwargs["force_cached"]` (and `force_query`), or add a `force_cached=True` case with the real helper?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] fix(charts): decouple annotation cache from the shared dataframe cache [superset]
Posted by "sadpandajoe (via GitHub)" <gi...@apache.org>.
sadpandajoe commented on code in PR #44406:
URL: https://github.com/apache/superset/pull/44406#discussion_r4169873660
##########
superset/common/query_context_processor.py:
##########
@@ -449,34 +481,189 @@ def query_cache_key(self, query_obj: QueryObject, **kwargs: Any) -> str | None:
)
return cache_key
- def _annotation_cache_context(self, query_obj: QueryObject) -> dict[str, Any]:
+ def annotation_cache_key(self, query_obj: QueryObject) -> str | None:
+ """
+ Cache key for this query's annotation-layer payload, or ``None`` when
+ the query has no annotation layers.
+
+ Annotation payloads are fetched under the requesting user's access
+ scope, which is a stricter security requirement than the dataframe
+ itself has. Keying them separately from :meth:`query_cache_key` keeps
+ that scoping from forcing every distinct viewer of an annotated chart
+ onto their own full copy of the (potentially much larger) shared
+ dataframe: users with the same access scope share this key too.
"""
- Cache-key material binding cached annotation data to its security
- context.
+ if not query_obj or not query_obj.annotation_layers:
+ return None
+ return self.query_cache_key(
+ query_obj, annotation_context=self._annotation_cache_context(query_obj)
+ )
- Annotation payloads are fetched per requesting user and stored on the
- same cache entry as the dataframe, so the key also binds the requesting
- user and, for chart-backed layers, the RLS clauses of the referenced
- chart's datasource.
+ def _annotation_cache_context(self, query_obj: QueryObject) -> dict[str, Any]:
+ """
+ Cache-key material binding annotation data to its security *scope* so
+ users with the same access share a cache entry and users with a
+ different scope β or no access β never read each other's data.
+
+ * NATIVE layers: the ``can_read`` permission on ``Annotation``, the
+ only user-dependent dimension of these global records.
+ * Chart-backed (``line``/``table``) layers: see
+ :meth:`_annotation_source_scope`.
"""
- source_rls: dict[str, list[str] | None] = {}
+ context: dict[str, Any] = {}
+
+ if any(
+ layer.get("sourceType") == "NATIVE" for layer in query_obj.annotation_layers
+ ):
+ context["annotation_read"] = security_manager.can_access(
+ "can_read", "Annotation"
+ )
+
+ source_scope: dict[str, Any] = {}
for layer in query_obj.annotation_layers:
if (
layer.get("sourceType")
not in ANNOTATION_SOURCE_TYPES_WITH_CHART_REFERENCE
):
continue
layer_value = layer.get("value")
+ source_scope[str(layer_value)] = self._annotation_source_scope(layer)
+ if source_scope:
+ context["source_scope"] = source_scope
+
+ return context
+
+ def _annotation_source_scope(self, layer: dict[str, Any]) -> dict[str, Any]:
+ """
+ Access and data-identity cache-key material for one chart-backed
+ annotation layer.
+
+ ``access`` keeps a user denied the referenced chart's datasource from
+ reading an authorized user's cached payload. When the chart has a
+ saved query context, this runs the *same* authorization path
+ :meth:`get_viz_annotation_data` executes
+ (``QueryContext.raise_for_access``) rather than the coarser
+ :meth:`SecurityManager.can_access_datasource` β a requester whose
+ access comes from a dashboard/viewer-promiscuous-mode bypass (which
+ depends on the chart's own saved ``form_data``, e.g. its
+ ``slice_id``/``dashboardId``) would otherwise still fail that coarser,
+ context-free check and collapse onto the same denied scope as a
+ genuinely unauthorized requester, letting the latter read the
+ former's cached payload. ``data_key`` is the annotation chart's own
+ query cache key(s) β derived from the same, override-applied query
+ objects actually executed (see :meth:`_apply_annotation_overrides`),
+ so it captures the datasource version, RLS clauses, and any per-user
+ Jinja/virtual-dataset RLS material exactly as the live fetch would,
+ including material an override only introduces at a finer grain.
+ Reusing this logic avoids re-deriving it and automatically inherits
+ any future correctness fixes made there.
+ """
+ layer_value = layer.get("value")
+ datasource = None
+ try:
chart = (
ChartDAO.find_by_id(layer_value) if layer_value is not None else None
)
- annotation_datasource = chart.datasource if chart else None
- source_rls[str(layer.get("value"))] = (
- security_manager.get_rls_cache_key(annotation_datasource)
- if annotation_datasource
- else None
+ # resolved_datasource, not datasource: the latter is pinned to
+ # table-backed datasources and resolves to None for a
+ # semantic-view-backed chart, which would otherwise collapse
+ # every requester onto the same {access: None, data_key: None}
+ # scope below regardless of their actual access.
+ datasource = chart.resolved_datasource if chart else None
+ if chart is None or datasource is None:
+ return {"access": None, "data_key": None}
Review Comment:
A chart can have its datasource_id cleared while its saved query context still targets a valid dataset, so this gives allowed and denied viewers the same unscoped annotation key; after an allowed viewer warms it, the denied viewer receives those rows without source authorization. Should unresolved scope, including derivation-error fallbacks, disable annotation-cache reuse rather than produce a shared identity?
##########
superset/common/query_context_processor.py:
##########
@@ -449,34 +481,189 @@ def query_cache_key(self, query_obj: QueryObject, **kwargs: Any) -> str | None:
)
return cache_key
- def _annotation_cache_context(self, query_obj: QueryObject) -> dict[str, Any]:
+ def annotation_cache_key(self, query_obj: QueryObject) -> str | None:
+ """
+ Cache key for this query's annotation-layer payload, or ``None`` when
+ the query has no annotation layers.
+
+ Annotation payloads are fetched under the requesting user's access
+ scope, which is a stricter security requirement than the dataframe
+ itself has. Keying them separately from :meth:`query_cache_key` keeps
+ that scoping from forcing every distinct viewer of an annotated chart
+ onto their own full copy of the (potentially much larger) shared
+ dataframe: users with the same access scope share this key too.
"""
- Cache-key material binding cached annotation data to its security
- context.
+ if not query_obj or not query_obj.annotation_layers:
+ return None
+ return self.query_cache_key(
+ query_obj, annotation_context=self._annotation_cache_context(query_obj)
+ )
- Annotation payloads are fetched per requesting user and stored on the
- same cache entry as the dataframe, so the key also binds the requesting
- user and, for chart-backed layers, the RLS clauses of the referenced
- chart's datasource.
+ def _annotation_cache_context(self, query_obj: QueryObject) -> dict[str, Any]:
+ """
+ Cache-key material binding annotation data to its security *scope* so
+ users with the same access share a cache entry and users with a
+ different scope β or no access β never read each other's data.
+
+ * NATIVE layers: the ``can_read`` permission on ``Annotation``, the
+ only user-dependent dimension of these global records.
+ * Chart-backed (``line``/``table``) layers: see
+ :meth:`_annotation_source_scope`.
"""
- source_rls: dict[str, list[str] | None] = {}
+ context: dict[str, Any] = {}
+
+ if any(
+ layer.get("sourceType") == "NATIVE" for layer in query_obj.annotation_layers
+ ):
+ context["annotation_read"] = security_manager.can_access(
+ "can_read", "Annotation"
+ )
+
+ source_scope: dict[str, Any] = {}
for layer in query_obj.annotation_layers:
if (
layer.get("sourceType")
not in ANNOTATION_SOURCE_TYPES_WITH_CHART_REFERENCE
):
continue
layer_value = layer.get("value")
+ source_scope[str(layer_value)] = self._annotation_source_scope(layer)
+ if source_scope:
+ context["source_scope"] = source_scope
+
+ return context
+
+ def _annotation_source_scope(self, layer: dict[str, Any]) -> dict[str, Any]:
+ """
+ Access and data-identity cache-key material for one chart-backed
+ annotation layer.
+
+ ``access`` keeps a user denied the referenced chart's datasource from
+ reading an authorized user's cached payload. When the chart has a
+ saved query context, this runs the *same* authorization path
+ :meth:`get_viz_annotation_data` executes
+ (``QueryContext.raise_for_access``) rather than the coarser
+ :meth:`SecurityManager.can_access_datasource` β a requester whose
+ access comes from a dashboard/viewer-promiscuous-mode bypass (which
+ depends on the chart's own saved ``form_data``, e.g. its
+ ``slice_id``/``dashboardId``) would otherwise still fail that coarser,
+ context-free check and collapse onto the same denied scope as a
+ genuinely unauthorized requester, letting the latter read the
+ former's cached payload. ``data_key`` is the annotation chart's own
+ query cache key(s) β derived from the same, override-applied query
+ objects actually executed (see :meth:`_apply_annotation_overrides`),
+ so it captures the datasource version, RLS clauses, and any per-user
+ Jinja/virtual-dataset RLS material exactly as the live fetch would,
+ including material an override only introduces at a finer grain.
+ Reusing this logic avoids re-deriving it and automatically inherits
+ any future correctness fixes made there.
+ """
+ layer_value = layer.get("value")
+ datasource = None
+ try:
chart = (
ChartDAO.find_by_id(layer_value) if layer_value is not None else None
)
- annotation_datasource = chart.datasource if chart else None
- source_rls[str(layer.get("value"))] = (
- security_manager.get_rls_cache_key(annotation_datasource)
- if annotation_datasource
- else None
+ # resolved_datasource, not datasource: the latter is pinned to
+ # table-backed datasources and resolves to None for a
+ # semantic-view-backed chart, which would otherwise collapse
+ # every requester onto the same {access: None, data_key: None}
+ # scope below regardless of their actual access.
+ datasource = chart.resolved_datasource if chart else None
+ if chart is None or datasource is None:
+ return {"access": None, "data_key": None}
+
+ annotation_query_context = chart.get_query_context()
+ if annotation_query_context is not None:
+ self._apply_annotation_overrides(annotation_query_context, layer)
+ try:
+ annotation_query_context.raise_for_access()
+ access: Any = True
+ except SupersetSecurityException:
+ access = False
+ data_key: Any = [
+ annotation_query_context.query_cache_key(query_object)
Review Comment:
For a saved samples context, execution clears metrics and selects all columns after this key is derived; an RLS template that calls current_user_id() only when metrics are absent therefore produces user-scoped rows under a shared annotation key. Should the scope key use the result handlerβs prepared query, matching what actually executes?
##########
superset/common/query_context_processor.py:
##########
@@ -449,34 +481,189 @@ def query_cache_key(self, query_obj: QueryObject, **kwargs: Any) -> str | None:
)
return cache_key
- def _annotation_cache_context(self, query_obj: QueryObject) -> dict[str, Any]:
+ def annotation_cache_key(self, query_obj: QueryObject) -> str | None:
+ """
+ Cache key for this query's annotation-layer payload, or ``None`` when
+ the query has no annotation layers.
+
+ Annotation payloads are fetched under the requesting user's access
+ scope, which is a stricter security requirement than the dataframe
+ itself has. Keying them separately from :meth:`query_cache_key` keeps
+ that scoping from forcing every distinct viewer of an annotated chart
+ onto their own full copy of the (potentially much larger) shared
+ dataframe: users with the same access scope share this key too.
"""
- Cache-key material binding cached annotation data to its security
- context.
+ if not query_obj or not query_obj.annotation_layers:
+ return None
+ return self.query_cache_key(
+ query_obj, annotation_context=self._annotation_cache_context(query_obj)
+ )
- Annotation payloads are fetched per requesting user and stored on the
- same cache entry as the dataframe, so the key also binds the requesting
- user and, for chart-backed layers, the RLS clauses of the referenced
- chart's datasource.
+ def _annotation_cache_context(self, query_obj: QueryObject) -> dict[str, Any]:
+ """
+ Cache-key material binding annotation data to its security *scope* so
+ users with the same access share a cache entry and users with a
+ different scope β or no access β never read each other's data.
+
+ * NATIVE layers: the ``can_read`` permission on ``Annotation``, the
+ only user-dependent dimension of these global records.
+ * Chart-backed (``line``/``table``) layers: see
+ :meth:`_annotation_source_scope`.
"""
- source_rls: dict[str, list[str] | None] = {}
+ context: dict[str, Any] = {}
+
+ if any(
+ layer.get("sourceType") == "NATIVE" for layer in query_obj.annotation_layers
+ ):
+ context["annotation_read"] = security_manager.can_access(
+ "can_read", "Annotation"
+ )
+
+ source_scope: dict[str, Any] = {}
for layer in query_obj.annotation_layers:
if (
layer.get("sourceType")
not in ANNOTATION_SOURCE_TYPES_WITH_CHART_REFERENCE
):
continue
layer_value = layer.get("value")
+ source_scope[str(layer_value)] = self._annotation_source_scope(layer)
Review Comment:
When two layers reference the same chart with different overrides, the later layer overwrites the earlier layerβs scope here; if only the earlier override activates per-user Jinja/RLS, the second viewer can receive the first viewerβs rows. Should the cache context retain every layerβs overridden scope rather than just one per chart ID?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] fix(charts): decouple annotation cache from the shared dataframe cache [superset]
Posted by "rusackas (via GitHub)" <gi...@apache.org>.
rusackas commented on code in PR #44406:
URL: https://github.com/apache/superset/pull/44406#discussion_r4344499888
##########
tests/unit_tests/common/test_query_context_processor.py:
##########
@@ -126,23 +135,471 @@ def processor(mock_query_context):
return processor
-def test_query_cache_key_binds_annotation_data_to_requesting_user(processor):
- """The cache key for annotated queries must differ per requesting user."""
+def test_annotation_cache_key_binds_native_annotation_read_scope(processor) -> None:
+ """The annotation cache key for NATIVE layers must differ when the
+ requester's ``can_read`` (Annotation) access differs -- not who they are."""
query_obj = MagicMock()
query_obj.annotation_layers = [{"sourceType": "NATIVE", "name": "a", "value": 1}]
+ # ``security_manager`` autodetects as an async spec under a bare
+ # ``patch()`` (its real object trips ``unittest.mock``'s coroutine
+ # inference), which would silently turn every attribute access into an
+ # ``AsyncMock`` returning a fresh unawaited coroutine per call -- always
+ # unequal to itself and never equal to a configured return value. Forcing
+ # ``new_callable=MagicMock`` keeps these synchronous, as the real object
+ # is.
+ with patch(
+ "superset.common.query_context_processor.security_manager",
+ new_callable=MagicMock,
+ ) as security_manager:
+ security_manager.can_access.side_effect = [True, False]
+ processor.annotation_cache_key(query_obj)
+ processor.annotation_cache_key(query_obj)
+ contexts = [
+ call.kwargs["annotation_context"] for call in query_obj.cache_key.call_args_list
+ ]
+ assert contexts[0] != contexts[1]
+
+
+def test_annotation_cache_key_shares_across_same_access_scope() -> None:
+ """Two distinct requesters (separate processor/query-object instances,
+ standing in for two different requests) with identical access scope must
+ produce identical annotation-context material. Reusing a single
+ processor/query_obj across both calls (as this test previously did)
+ would pass trivially regardless of whether the key is scope-based or
+ identity-based, since nothing about "who's asking" would ever vary."""
+ layer = {"sourceType": "NATIVE", "name": "a", "value": 1}
+ processor_a = QueryContextProcessor(MagicMock())
+ processor_b = QueryContextProcessor(MagicMock())
+ query_obj_a = MagicMock(annotation_layers=[layer])
+ query_obj_b = MagicMock(annotation_layers=[layer])
+
+ # The requesting user differs between the two calls. Binding user identity
+ # into the context (by name, from either module) would make the contexts
+ # diverge, so equality below proves the key is scope-based.
with (
+ patch(
+ "superset.common.query_context_processor.security_manager",
+ new_callable=MagicMock,
+ ) as security_manager,
patch(
"superset.common.query_context_processor.get_user_id",
+ create=True,
side_effect=[1, 2],
),
- patch("superset.common.query_context_processor.security_manager"),
+ patch("superset.utils.core.get_user_id", side_effect=[1, 2]),
+ ):
+ security_manager.can_access.return_value = True
+ context_a = processor_a._annotation_cache_context(query_obj_a)
+ context_b = processor_b._annotation_cache_context(query_obj_b)
+
+ assert context_a == context_b
+
+
+def test_query_cache_key_does_not_bind_annotation_scope(processor) -> None:
+ """The dataframe cache key must stay shared across viewers of the same
+ chart, even when the query has annotation layers β only the separate
+ annotation cache key (see above) carries access-scope material."""
+ query_obj = MagicMock()
+ query_obj.annotation_layers = [{"sourceType": "NATIVE", "name": "a", "value": 1}]
+ with patch(
+ "superset.common.query_context_processor.security_manager",
+ new_callable=MagicMock,
):
processor.query_cache_key(query_obj)
processor.query_cache_key(query_obj)
- contexts = [
- call.kwargs["annotation_context"] for call in query_obj.cache_key.call_args_list
+ for call in query_obj.cache_key.call_args_list:
+ assert "annotation_context" not in call.kwargs
+
+
+@pytest.fixture
+def mock_annotation_chart() -> Iterator[MagicMock]:
+ """A found chart, wired as the referenced chart for
+ ``_annotation_source_scope`` tests -- factors out the repeated
+ ``ChartDAO.find_by_id`` patch those tests all need."""
+ chart = MagicMock()
+ with patch(
+ "superset.common.query_context_processor.ChartDAO.find_by_id",
+ return_value=chart,
+ ):
+ yield chart
+
+
+def test_annotation_source_scope_binds_datasource_access(
+ processor: QueryContextProcessor, mock_annotation_chart: MagicMock
+) -> None:
+ """A chart-backed annotation layer's scope must differ when the
+ requester's access to the referenced datasource differs."""
+ mock_annotation_chart.get_query_context.return_value = None
+ with patch(
+ "superset.common.query_context_processor.security_manager",
+ new_callable=MagicMock,
+ ) as security_manager:
+ security_manager.can_access_datasource.side_effect = [True, False]
+ security_manager.get_rls_cache_key.return_value = []
+ scope_a = processor._annotation_source_scope({"value": 1})
+ scope_b = processor._annotation_source_scope({"value": 1})
+ assert scope_a is not None
+ assert scope_b is not None
+ assert scope_a != scope_b
+ assert scope_a["access"] is True
+ assert scope_b["access"] is False
+
+
+def test_annotation_source_scope_reuses_referenced_chart_cache_key(
+ processor: QueryContextProcessor, mock_annotation_chart: MagicMock
+) -> None:
+ """When the referenced chart has a saved query context, its own cache
+ key(s) -- covering RLS and per-user Jinja/virtual-dataset material -- are
+ reused rather than re-derived."""
+ mock_query_object: MagicMock = MagicMock()
+ mock_query_context: MagicMock = MagicMock()
+ mock_query_context.queries = [mock_query_object]
+ mock_query_context.query_cache_key.return_value = "referenced-chart-key"
+ mock_annotation_chart.get_query_context.return_value = mock_query_context
+ with patch(
+ "superset.common.query_context_processor.security_manager",
+ new_callable=MagicMock,
+ ) as security_manager:
+ security_manager.can_access_datasource.return_value = True
+ scope = processor._annotation_source_scope({"value": 1})
+ assert scope == {"access": True, "data_key": ["referenced-chart-key"]}
+ mock_query_context.query_cache_key.assert_called_once_with(mock_query_object)
+
+
+def test_annotation_source_scope_keys_on_result_type_prepared_query(
+ processor: QueryContextProcessor, mock_annotation_chart: MagicMock
+) -> None:
+ """A referenced chart saved with a samples result type executes a
+ rewritten query (metrics cleared, every column selected), so its key
+ must be derived from that prepared query, not the saved one -- a Jinja
+ clause that only renders user-specific material without metrics would
+ otherwise give two viewers one key over different rows."""
+ from superset.common.query_object import QueryObject
+
+ mock_datasource = MagicMock()
+ mock_datasource.columns = []
+ saved_query = QueryObject(
+ datasource=mock_datasource, metrics=["count"], columns=["col1"]
+ )
+ mock_query_context = MagicMock()
+ mock_query_context.result_type = ChartDataResultType.SAMPLES
Review Comment:
Added `test_annotation_source_scope_keys_on_query_level_samples_result_type`: a `FULL` context with a query-level `SAMPLES` result type, asserting `data_key` comes from the cleared-metrics query.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] fix(charts): decouple annotation cache from the shared dataframe cache [superset]
Posted by "sadpandajoe (via GitHub)" <gi...@apache.org>.
sadpandajoe commented on code in PR #44406:
URL: https://github.com/apache/superset/pull/44406#discussion_r4323408810
##########
tests/unit_tests/common/test_query_context_processor.py:
##########
@@ -3167,11 +3915,211 @@ def payload(context: QueryContext, **kwargs: Any) -> dict[str, Any]:
datasource="native-1", annotation_context=context
)
assert current_key != legacy_key
- assert context == {
- **legacy_context,
- "semantic_filter_protocol": "semantic-null-filters-v1",
- "source_versions": {"7": view.metadata_generation},
- }
+ assert context is not None
+ assert context["source_scope"][0]["semantic_filter_protocol"] == (
+ "semantic-null-filters-v1"
+ )
+ assert context["source_scope"][0]["metadata_version"] == view.metadata_generation
+
+
+def _annotated_payload_fixture(
+ *, status: str = "success"
+) -> tuple[QueryContextProcessor, Any, MagicMock]:
+ """A processor, annotated query object, and a dataframe cache hit."""
+ from superset.common.query_object import QueryObject
+
+ mock_query_context = MagicMock()
+ mock_query_context.force = False
+ mock_datasource = MagicMock()
+ mock_datasource.column_names = ["col1"]
+ processor = QueryContextProcessor(mock_query_context)
+ processor._qc_datasource = mock_datasource
+ query_obj = QueryObject(
+ datasource=mock_datasource,
+ columns=["col1"],
+ annotation_layers=[
+ {
+ "annotationType": "EVENT",
+ "sourceType": "NATIVE",
+ "name": "a",
+ "value": 1,
+ }
+ ],
+ )
+ cache = MagicMock()
+ cache.is_loaded = True
+ cache.applied_filter_columns = ["col1"]
+ cache.df = pd.DataFrame({"col1": [1, 2, 3]})
+ cache.query = ""
+ cache.status = status
+ cache.error_message = "dataframe failed" if status == QueryStatus.FAILED else None
+ cache.is_cached = True
+ cache.applied_template_filters = []
+ cache.rejected_filter_columns = []
+ cache.bq_memory_limited = False
+ cache.result_persisted = False
+ return processor, query_obj, cache
+
+
+def _run_annotated_payload(
+ processor: QueryContextProcessor,
+ query_obj: Any,
+ cache: MagicMock,
+ *,
+ force_cached: bool = False,
+) -> Any:
+ with (
+ patch(
+ "superset.common.query_context_processor.QueryCacheManager"
+ ) as mock_cache_manager,
+ patch.object(query_obj, "validate", return_value=None),
+ patch.object(processor, "query_cache_key", return_value="df-key"),
+ patch.object(processor, "annotation_cache_key", return_value="ann-key"),
+ patch.object(processor, "get_cache_timeout", return_value=3600),
+ ):
+ mock_cache_manager.get.return_value = cache
+ return processor.get_df_payload_result(query_obj, force_cached=force_cached)
+
+
+def test_get_df_payload_result_skips_annotations_when_dataframe_failed() -> None:
+ """A failed dataframe keeps its own error and never triggers the fetch."""
+ processor, query_obj, cache = _annotated_payload_fixture(status=QueryStatus.FAILED)
+ with patch.object(processor, "_get_annotation_data_cached") as mock_annotation:
+ result = _run_annotated_payload(processor, query_obj, cache)
+ mock_annotation.assert_not_called()
+ assert result.payload["annotation_data"] == {}
+ assert result.payload["error"] == "dataframe failed"
+
+
+def test_get_df_payload_result_force_cached_misses_without_annotation_entry() -> None:
+ """Through the real helper, ``force_cached`` with no annotation entry
+ raises ``CacheLoadError`` instead of running the annotation SQL."""
+ processor, query_obj, cache = _annotated_payload_fixture()
+ with (
+ patch("superset.common.query_context_processor.cache_manager") as cm,
+ patch.object(processor, "get_annotation_data") as mock_live,
+ ):
+ cm.data_cache.get.return_value = None
+ with pytest.raises(CacheLoadError):
+ _run_annotated_payload(processor, query_obj, cache, force_cached=True)
+ mock_live.assert_not_called()
+
+
+def test_get_df_payload_result_annotation_failure_marks_payload_failed() -> None:
+ """An annotation error on a good dataframe fails the payload."""
+ processor, query_obj, cache = _annotated_payload_fixture()
+ with (
+ patch("superset.common.query_context_processor.cache_manager") as cm,
+ patch.object(
+ processor,
+ "get_annotation_data",
+ side_effect=QueryObjectValidationError("annotation boom"),
+ ),
+ ):
+ cm.data_cache.get.return_value = None
+ result = _run_annotated_payload(processor, query_obj, cache)
+ assert cache.status == QueryStatus.FAILED
+ assert result.payload["error"] == "annotation boom"
Review Comment:
This assertion also passes if the annotation catch goes back to storing a plain `str(ex)`, because `QueryObjectValidationMessage` is a `str` subclass and `==` ignores the type. Reverting the fix would leave this test green while a deleted annotation layer surfaces as `ChartDataQueryFailedError` instead of the 400 validation error. Could this also assert `isinstance(result.payload["error"], QueryObjectValidationMessage)` (and that the cause is retained), as `test_get_df_payload_keeps_dataset_validation_type` does for the dataframe path?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] fix(charts): decouple annotation cache from the shared dataframe cache [superset]
Posted by "sadpandajoe (via GitHub)" <gi...@apache.org>.
sadpandajoe commented on code in PR #44406:
URL: https://github.com/apache/superset/pull/44406#discussion_r4322061779
##########
superset/common/query_context_processor.py:
##########
@@ -285,6 +279,18 @@ def get_df_payload_result(
# decision as the result cache, not the datasource-level default.
query_obj.force_query = force_query
query_obj.cache_timeout = timeout
+ # Resolved separately from the dataframe's force_query, against its own
+ # annotation_key. A forced GTF refresh's nonce marker is per (nonce,
+ # cache_key): the dataframe's marker is keyed on cache_key, so once it's
+ # set, force_query goes False for every subsequent request carrying that
+ # nonce, including a request for a *different* annotation access scope
+ # (its own annotation_key) that has never actually forced-refreshed its
+ # own entry. Reusing the dataframe's force_query there would read that
+ # scope's existing annotation cache entry instead of refreshing it.
+ annotation_force_query = (
+ self._resolve_forced_query(query_obj, annotation_key)
+ or timeout == CACHE_DISABLED_TIMEOUT
Review Comment:
The annotation-specific `timeout == CACHE_DISABLED_TIMEOUT` guard has no test: the annotation payload tests all patch `get_cache_timeout` to 3600/123/300, so deleting this clause would leave them green and a chart with caching disabled (`-1`) and a previously warmed annotation entry would serve that stale entry instead of fetching live. Could a `get_df_payload_result` test use `get_cache_timeout=-1` with `force=False` and a warm annotation entry, and assert the live annotation fetch runs and its result replaces the stale value?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org