You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@superset.apache.org by "betodealmeida (via GitHub)" <gi...@apache.org> on 2026/05/18 13:47:23 UTC

[PR] feat(semantic-layers): containment-aware query cache [superset]

betodealmeida opened a new pull request, #40221:
URL: https://github.com/apache/superset/pull/40221

   <!---
   Please write the PR title following the conventions at https://www.conventionalcommits.org/en/v1.0.0/
   Example:
   fix(dashboard): load charts correctly
   -->
   
   ### SUMMARY
   <!--- Describe the change below, including rationale and design decisions -->
   
   Adds a smart, containment-aware cache for the semantic-layer query path. The cache understands that one query's result can satisfy *another* query whose filters/limit/dimensions/metrics are narrower — and serves those subsequent requests from a locally post-processed `DataFrame` instead of hitting the warehouse.
   
   In dashboards and Explore the win is concrete: changing a filter range, shrinking a top-N, dropping a metric, or rolling up a dimension no longer round-trips to the database when a broader prior result is already cached.
   
   A small UI affordance in Explore signals that a chart was served from the semantic-layer cache (distinct from the existing per-query result cache).
   
   #### How it works
   
   ##### Cache layout
   
   * **Index**: one Redis key per `(view UUID, changed_on)` — `sv:idx:<uuid>:<changed_on_iso>`. Holds a list of `CachedEntry` descriptors. Bumping the view's `changed_on` (schema/definition change) invalidates the whole bucket atomically.
   * **Value**: one Redis key per canonicalized query — `sv:val:<uuid>:<changed_on_iso>:<query_digest>`. Holds the
     `SemanticResult` (Arrow table + provenance).
   * **TTL**: takes `SemanticView.cache_timeout` first, then the global `DATA_CACHE_CONFIG.CACHE_DEFAULT_TIMEOUT`.
   
   `CachedEntry` is a frozen dataclass that captures filters, dimension keys, metric IDs, limit/offset/order/group-limit, and the value key. It is the unit the lookup walks; the value is fetched lazily once a candidate is chosen.
   
   ##### Reuse modes
   
   When a query lands, `try_serve_from_cache` walks the bucket and asks `can_satisfy(entry, query)` for each entry. The check returns one of three `ReuseMode`s:
   
   | Mode      | When                                                | Post-processing                                   |
   | --------- | --------------------------------------------------- | ------------------------------------------------- |
   | `EXACT`   | Same dims, same metrics                             | Apply leftover filters / limit / order            |
   | `PROJECT` | Same dims; cached metrics ⊃ new metrics             | Drop the unused metric columns                    |
   | `ROLLUP`  | Cached dims ⊃ new dims                              | `groupby` re-aggregate the cached `DataFrame`     |
   
   In all three modes, cached metrics must be a non-strict superset of the new query's metrics. `ROLLUP` additionally requires the new metrics to use additive aggregations (`SUM`, `COUNT`, `MIN`, `MAX`). `PROJECT` has no additive constraint because row granularity is preserved — non-additive aggregates like `AVG` survive unchanged.
   
   When multiple entries satisfy the query, we rank them `EXACT < PROJECT < ROLLUP` and serve the cheapest. If the preferred entry's value has been evicted between index lookup and value fetch, we fall back to the next-best candidate.
   
   ##### Filter containment
   
   For `WHERE` filters, each cached predicate must be *implied by* some new predicate on the same column (cached cannot have hidden rows the new query needs). Conversely, new predicates not covered by the cache become "leftovers" and are reapplied to the cached DataFrame post-fetch, provided they reference columns that are still in the cached projection.
   
   The implication helpers cover ranges (`>`, `>=`, `<`, `<=`), `EQUALS`, `IN`/`NOT_IN`, `IS_NULL`/`IS_NOT_NULL`, and exact-match `LIKE`. `HAVING` and `ADHOC` filters require exact-set equality — without parsing them, we cannot prove containment.
   
   Filter values are coerced against the dimension's pyarrow type (`_coerce_filter_value`) before containment checks. This fixes a class of false-miss bugs where a UI sends `"1984"` as a string against an integer dimension and the cache fails to recognize that `a >= 1984` is a subset of `a >= 1982`.
   
   ##### Top-N safety
   
   When a cached entry has a `limit`, we only reuse it if the new query has the same order and a tighter limit (EXACT and PROJECT paths). For `ROLLUP`, we require the cached result to be *known complete* — if `payload.results.num_rows < entry.limit`, the source had fewer rows than the cap, so the rollup is over the whole population. Otherwise the cache is skipped (the index entry is preserved; only that reuse is blocked).
   
   ##### Capacity & eviction
   
   The per-view bucket caps at `MAX_ENTRIES_PER_VIEW = 128` (oldest entries dropped first). Each value lives under its own Redis key with its own TTL, so Redis-level LRU and TTLs do the actual eviction; the index is kept consistent by pruning entries whose values are gone on every lookup.
   
   ##### What is *not* cached
   
   * `offset != 0`: pagination beyond the first page is not reused (no efficient way to recover middle pages from a broader cached result).
   * `is_rowcount` and `force_query=True`: bypass the cache entirely.
   * Cross-view queries: cache keys are scoped per `(view, changed_on)`.
   
   ##### Provenance
   
   Every served result records the cache hit in its `requests` list as a `SemanticRequest(type="cache", definition=...)` with a mode-specific note — useful for debugging and for the Explore indicator to show "semantic smart cache" provenance.
   
   #### Frontend
   
   `CachedLabel` and its tooltip learn a new `cacheSource` prop: `"query"` (legacy) or `"semantic"`. `ChartPills` in Explore passes `"semantic"` when the chart result was served from the semantic-layer cache, distinguishing it from the older per-query result cache.
   
   #### Future work
   
   * Surface a "smart-cache disabled" toggle in Explore for debugging.
   * Investigate raising `MAX_ENTRIES_PER_VIEW` per-view via the `SemanticView` schema.
   * Add metrics/telemetry for hit rate, mode breakdown, and post-processing latency so we can tune containment heuristics with real data.
   
   
   ### BEFORE/AFTER SCREENSHOTS OR ANIMATED GIF
   <!--- Skip this if not applicable -->
   
   ### TESTING INSTRUCTIONS
   <!--- Required! What steps can be taken to manually verify the changes? -->
   
   * **`tests/unit_tests/semantic_layers/cache_test.py`** — implication helpers, containment edge cases, mode selection, limit/order/offset behavior, and post-processing for each `ReuseMode`.
   * **`tests/unit_tests/semantic_layers/cache_integration_test.py`** — end-to-end coverage via `mapper.get_results` with an in-memory cache fixture: narrower-filter reuse, projection rollup, metric-subset (PROJECT) hits, the EXACT-vs-ROLLUP preference, and the eviction fallback path.
   * **`tests/unit_tests/semantic_layers/mapper_test.py`** — filter coercion regressions.
   
   ### ADDITIONAL INFORMATION
   <!--- Check any relevant boxes with "x" -->
   <!--- HINT: Include "Fixes #nnn" if you are fixing an existing issue -->
   - [ ] Has associated issue:
   - [ ] Required feature flags:
   - [ ] Changes UI
   - [ ] Includes DB Migration (follow approval process in [SIP-59](https://github.com/apache/superset/issues/13351))
     - [ ] Migration is atomic, supports rollback & is backwards-compatible
     - [ ] Confirm DB migration upgrade and downgrade tested
     - [ ] Runtime estimates and downtime expectations provided
   - [ ] Introduces new feature or API
   - [ ] Removes existing feature or API
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org


Re: [PR] feat(semantic-layers): containment-aware query cache [superset]

Posted by "mikebridge (via GitHub)" <gi...@apache.org>.
mikebridge commented on PR #40221:
URL: https://github.com/apache/superset/pull/40221#issuecomment-4897979881

   Triage of the seven open bot threads on this PR (verified where load-bearing, via Claude):
   
   1. **`query_object.py` — codeant "Architect HIGH" on `force_query` in the cache key: VERIFIED REAL** and fixed in **#41825** (into this branch). A forced refresh was writing its fresh result under a different cache key while ordinary requests kept serving the stale entry — affects every chart on the legacy path, not just semantic layers. One-line fix + regression test.
   2. **`TooltipContent.test.tsx` — bito's `data-testid` mismatch: FALSE POSITIVE.** The repo's single jest config covers `packages/` (testRegex) and loads `spec/helpers/setup.ts`, which sets `testIdAttribute: 'data-test'` — so `getByTestId` matches `data-test` attributes correctly. Safe to resolve.
   3. **`cache.py` broad-except suggestions (bito)**: the blanket catches are the file's deliberate defensive pattern (housekeeping must never fail a request) and carry pragma annotations; also reviewed as intentional in the #41824 review rounds. Safe to resolve as by-design.
   4. **Select.tsx ×2 (counts concatenated outside `t()`) and the JSON Forms `placeholderText` translation**: legitimate minor i18n nits — worth a small cleanup commit but not blocking.
   5. **`TooltipContent.tsx` duplication (two near-identical `cacheSource` conditionals)**: fair nit, one small helper.
   
   Separately: **#41824** (also into this branch) fixes the twice-flagged `store_result` index race, hardened through three review rounds + live-Redis validation — details in that PR's comment thread.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org


Re: [PR] feat(semantic-layers): containment-aware query cache [superset]

Posted by "michael-s-molina (via GitHub)" <gi...@apache.org>.
michael-s-molina commented on PR #40221:
URL: https://github.com/apache/superset/pull/40221#issuecomment-4652544343

   @betodealmeida Thank you for the PR. This cache is a good default for lightweight implementations, but it would be great to also provide an opt-out mechanism for semantic layers that have their own query optimizer, result cache, and data-aware invalidation (e.g. Minerva) — to avoid Superset serving stale data or producing incorrect re-aggregations from metrics that encode complex business logic.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org


Re: [PR] feat(semantic-layers): containment-aware query cache [superset]

Posted by "netlify[bot] (via GitHub)" <gi...@apache.org>.
netlify[bot] commented on PR #40221:
URL: https://github.com/apache/superset/pull/40221#issuecomment-4478341506

   ### <span aria-hidden="true">✅</span> Deploy Preview for *superset-docs-preview* ready!
   
   
   |  Name | Link |
   |:-:|------------------------|
   |<span aria-hidden="true">🔨</span> Latest commit | 411d0401309e91b491a5f635a3cfca7147164c72 |
   |<span aria-hidden="true">🔍</span> Latest deploy log | https://app.netlify.com/projects/superset-docs-preview/deploys/6a0b186c2b580d00084e95cc |
   |<span aria-hidden="true">😎</span> Deploy Preview | [https://deploy-preview-40221--superset-docs-preview.netlify.app](https://deploy-preview-40221--superset-docs-preview.netlify.app) |
   |<span aria-hidden="true">📱</span> Preview on mobile | <details><summary> Toggle QR Code... </summary><br /><br />![QR Code](https://app.netlify.com/qr-code/eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJ1cmwiOiJodHRwczovL2RlcGxveS1wcmV2aWV3LTQwMjIxLS1zdXBlcnNldC1kb2NzLXByZXZpZXcubmV0bGlmeS5hcHAifQ.BAVl17WTZx4TRZW8caiLWDKlxOKKO85nPjvJKOqVpzQ)<br /><br />_Use your smartphone camera to open QR code link._</details> |
   |<span aria-hidden="true">🤖</span> Make changes | [Run an agent on this branch](https://app.netlify.com/projects/superset-docs-preview/agent-runs#q1YqKMrPLShRslJS0lFKKkrMS84AsotzdJMTkzNSgWIpqQU5-ZWeKUBRs0SDJEMLs2SjJFMLgxQDAwMLk1RL0-RkpVoA) |
   ---
   <!-- [superset-docs-preview Preview](https://deploy-preview-40221--superset-docs-preview.netlify.app) -->
   _To edit notification comments on pull requests, go to your [Netlify project configuration](https://app.netlify.com/projects/superset-docs-preview/configuration/notifications#deploy-notifications)._


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org


Re: [PR] feat(semantic-layers): containment-aware query cache [superset]

Posted by "mikebridge (via GitHub)" <gi...@apache.org>.
mikebridge commented on PR #40221:
URL: https://github.com/apache/superset/pull/40221#issuecomment-4905817027

   **CI unblock for this branch and its stack: #41856.** While validating the stacked PRs we found `sl-cache`'s own `unit-tests (current)` is red on the workflow's 100% coverage gate for `superset/semantic_layers/` (the branch sits at 86.44%), and every stacked PR inherits that red. **#41856** (into this branch) closes the gap to 100.00% — 499 tests, mostly table-driven over the untested implication/mask/coercion edges — after which this branch's job and #41824/#41825/#41826 all go green.
   
   Writing those tests surfaced three things worth this thread's attention:
   
   1. **Real bug, fixed in #41856**: negative leftover masks (`NOT_EQUALS`/`NOT_IN`/`NOT_LIKE`) kept NULL rows — pandas `NaN != x` is True, SQL three-valued logic excludes it — so cache-served results could contain rows the warehouse would have dropped.
   2. **Semantics divergence, flagged**: `_sql_like_to_regex` has no escape support — `LIKE '100\%'` wildcards the `%`, unlike e.g. PostgreSQL's default backslash escape — so leftover `LIKE` masks can diverge from warehouse behavior for escaped patterns. (This is the concrete version of the earlier `_sql_like_to_regex` review note.)
   3. **Dead code**: two fallthrough `return False` lines in `_implies_range` are unreachable (both same-direction range-op combinations are exhaustively enumerated above them) — pragma'd with justification in #41856; deleting them is equally fine.
   
   Positive note for confidence in the design: the full pairwise implication matrix (every `IS_NULL`/`EQUALS`/`IN`/`NOT_IN`/range cross-combination) passed against SQL semantics on the first run — the containment logic's core is sound; the gaps were at the mask/NULL boundary.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org


Re: [PR] feat(semantic-layers): containment-aware query cache [superset]

Posted by "codeant-ai-for-open-source[bot] (via GitHub)" <gi...@apache.org>.
codeant-ai-for-open-source[bot] commented on code in PR #40221:
URL: https://github.com/apache/superset/pull/40221#discussion_r3268563867


##########
superset-frontend/packages/superset-ui-core/src/components/Select/Select.tsx:
##########
@@ -519,7 +519,8 @@ const Select = forwardRef(
               handleSelectAll();
             }}
           >
-            {t('Select all')} {`(${formatNumber('SMART_NUMBER', bulkSelectCounts.selectable)})`}
+            {t('Select all')}{' '}
+            {`(${formatNumber('SMART_NUMBER', bulkSelectCounts.selectable)})`}

Review Comment:
   **Suggestion:** The selectable count is rendered outside the translation string, so translators cannot reorder or localize the full label (text + number) correctly for languages with different grammar. Put the count inside a single `t(...)` message with a placeholder so the whole phrase is translatable. [code quality]
   
   <details>
   <summary><b>Severity Level:</b> Major ⚠️</summary>
   
   ```mdx
   - ⚠️ Bulk select label poorly localized in non-English locales.
   - ⚠️ Translators cannot reorder count relative to label text.
   ```
   </details>
   <details>
   <summary><b>Steps of Reproduction ✅ </b></summary>
   
   ```mdx
   1. Open `superset-frontend/packages/superset-ui-core/src/components/Select/Select.tsx` and
   locate the `bulkSelectComponent` `useMemo` defined near the bottom of the file; within
   this JSX block, at diff lines 522–523, the label is rendered as `{t('Select all')}{' '}`
   on one line and `{`(${formatNumber('SMART_NUMBER', bulkSelectCounts.selectable)})`}` on
   the next.
   
   2. In the same file, at diff lines 552–565 (confirmed via BulkRead), `popupRender` passes
   `selectAllEnabled ? bulkSelectComponent : undefined` into `dropDownRenderHelper`, so
   whenever `selectAllEnabled` (computed earlier in this file around lines 11–23 of the
   BulkRead snippet) is true, the dropdown renders this combined label to the user.
   
   3. Run the frontend with a non-English locale enabled so that `t('Select all')` resolves
   via the i18n catalogs; observe that translators only see and can translate the bare string
   `"Select all"` because the numeric count and parentheses `(${...})` are added in code and
   are not part of the translation key.
   
   4. Observe in the UI that the rendered text always follows the hardcoded English-oriented
   structure `"<translated Select all> (<count>)"`; translators cannot change the position of
   the count, add language-specific glue words, or adapt parentheses because the count is
   concatenated outside `t(...)`, demonstrating a localization defect tied directly to the
   code at lines 522–523.
   ```
   </details>
   
   [Fix in Cursor](https://app.codeant.ai/fix-in-ide?tool=cursor&prompt_id=ac76556a642b4f9a89dcb11e593b9565&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset) | [Fix in VSCode Claude](https://app.codeant.ai/fix-in-ide?tool=vscode-claude&prompt_id=ac76556a642b4f9a89dcb11e593b9565&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset)
   
   *(Use Cmd/Ctrl + Click for best experience)*
   <details>
   <summary><b>Prompt for AI Agent 🤖 </b></summary>
   
   ```mdx
   This is a comment left during a code review.
   
   **Path:** superset-frontend/packages/superset-ui-core/src/components/Select/Select.tsx
   **Line:** 522:523
   **Comment:**
   	*Code Quality: The selectable count is rendered outside the translation string, so translators cannot reorder or localize the full label (text + number) correctly for languages with different grammar. Put the count inside a single `t(...)` message with a placeholder so the whole phrase is translatable.
   
   Validate the correctness of the flagged issue. If correct, How can I resolve this? If you propose a fix, implement it and please make it concise.
   Once fix is implemented, also check other comments on the same PR, and ask user if the user wants to fix the rest of the comments as well. if said yes, then fetch all the comments validate the correctness and implement a minimal fix
   ```
   </details>
   <a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F40221&comment_hash=ee1fcddfcb11b86ac0770ef78f2e300519395106d4189b0749cda5b616022748&reaction=like'>👍</a> | <a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F40221&comment_hash=ee1fcddfcb11b86ac0770ef78f2e300519395106d4189b0749cda5b616022748&reaction=dislike'>👎</a>



##########
superset-frontend/packages/superset-ui-core/src/components/Select/Select.tsx:
##########
@@ -536,7 +537,8 @@ const Select = forwardRef(
               handleDeselectAll();
             }}
           >
-            {t('Clear')} {`(${formatNumber('SMART_NUMBER', bulkSelectCounts.deselectable)})`}
+            {t('Clear')}{' '}
+            {`(${formatNumber('SMART_NUMBER', bulkSelectCounts.deselectable)})`}

Review Comment:
   **Suggestion:** The clear-action count is concatenated outside the i18n message, which prevents proper localization of the complete phrase in non-English locales. Use one translated string with an interpolation placeholder for the formatted count. [code quality]
   
   <details>
   <summary><b>Severity Level:</b> Major ⚠️</summary>
   
   ```mdx
   - ⚠️ Bulk clear label not fully localizable in other languages.
   - ⚠️ Translators constrained by hardcoded text-count concatenation.
   ```
   </details>
   <details>
   <summary><b>Steps of Reproduction ✅ </b></summary>
   
   ```mdx
   1. In `superset-frontend/packages/superset-ui-core/src/components/Select/Select.tsx`,
   within the same `bulkSelectComponent` `useMemo` block used for bulk actions, locate the
   second button for the clear action; at diff lines 540–541 it renders `{t('Clear')}{' '}`
   on one line and `{`(${formatNumber('SMART_NUMBER', bulkSelectCounts.deselectable)})`}` on
   the next.
   
   2. As confirmed by BulkRead, `popupRender` (diff lines 552–565) passes `selectAllEnabled ?
   bulkSelectComponent : undefined` into `dropDownRenderHelper`, so whenever the bulk actions
   footer is shown, this clear button (with its split label) is rendered in the dropdown UI.
   
   3. Enable a non-English locale so that `t('Clear')` resolves via the i18n catalogs; note
   that the translation system only sees `"Clear"` as the message key, while the deselectable
   count and parentheses are added separately via `formatNumber('SMART_NUMBER',
   bulkSelectCounts.deselectable)`.
   
   4. Observe in the rendered UI that the clear action label always appears as `"<translated
   Clear> (<count>)"`, and translators cannot move the count before the word, change the
   punctuation, or integrate the number into the localized phrase because the count is not
   part of the `t(...)` message, demonstrating the same localization limitation at lines
   540–541.
   ```
   </details>
   
   [Fix in Cursor](https://app.codeant.ai/fix-in-ide?tool=cursor&prompt_id=033b73dc4d0a4be58bb9c0da17015f73&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset) | [Fix in VSCode Claude](https://app.codeant.ai/fix-in-ide?tool=vscode-claude&prompt_id=033b73dc4d0a4be58bb9c0da17015f73&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset)
   
   *(Use Cmd/Ctrl + Click for best experience)*
   <details>
   <summary><b>Prompt for AI Agent 🤖 </b></summary>
   
   ```mdx
   This is a comment left during a code review.
   
   **Path:** superset-frontend/packages/superset-ui-core/src/components/Select/Select.tsx
   **Line:** 540:541
   **Comment:**
   	*Code Quality: The clear-action count is concatenated outside the i18n message, which prevents proper localization of the complete phrase in non-English locales. Use one translated string with an interpolation placeholder for the formatted count.
   
   Validate the correctness of the flagged issue. If correct, How can I resolve this? If you propose a fix, implement it and please make it concise.
   Once fix is implemented, also check other comments on the same PR, and ask user if the user wants to fix the rest of the comments as well. if said yes, then fetch all the comments validate the correctness and implement a minimal fix
   ```
   </details>
   <a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F40221&comment_hash=e1d03b6af038639fd073187421654d568a449f161f2f50759e63d8156861f118&reaction=like'>👍</a> | <a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F40221&comment_hash=e1d03b6af038639fd073187421654d568a449f161f2f50759e63d8156861f118&reaction=dislike'>👎</a>



##########
superset/common/query_object.py:
##########
@@ -128,6 +129,7 @@ def __init__(  # pylint: disable=too-many-locals, too-many-arguments
         post_processing: list[dict[str, Any] | None] | None = None,
         row_limit: int | None = None,
         row_offset: int | None = None,
+        force_query: bool = False,
         series_columns: list[Column] | None = None,

Review Comment:
   **🟠 Architect Review — HIGH**
   
   QueryObject.to_dict() now includes the execution-control flag force_query, and cache_key() hashes that dict directly; this makes forced-refresh requests use a different cache key than identical non-forced requests, so a user-triggered refresh populates a separate cache entry while normal follow-up requests continue serving stale cached data under the old key.
   
   **Suggestion:** Keep force_query as an execution-control flag but exclude it from cache key materialization (e.g., remove it from the dict inside cache_key() before hashing, or build the cache dict without it) so forced refreshes update the canonical cache entry shared with normal requests.
   
   
   [Fix in Cursor](https://app.codeant.ai/fix-in-ide?tool=cursor&prompt_id=5f8546beb26f4b6a827a8f269dfd5a41&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset) | [Fix in VSCode Claude](https://app.codeant.ai/fix-in-ide?tool=vscode-claude&prompt_id=5f8546beb26f4b6a827a8f269dfd5a41&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset)
   
   *(Use Cmd/Ctrl + Click for best experience)*
   <details>
   <summary><b>Prompt for AI Agent 🤖 </b></summary>
   
   ```mdx
   This is an **Architect / Logical Review** comment left during a code review. These reviews are first-class, important findings — not optional suggestions. Do NOT dismiss this as a 'big architectural change' just because the title says architect review; most of these can be resolved with a small, localized fix once the intent is understood.
   
   **Path:** superset/common/query_object.py
   **Line:** 133:179
   **Comment:**
   	*HIGH: QueryObject.to_dict() now includes the execution-control flag force_query, and cache_key() hashes that dict directly; this makes forced-refresh requests use a different cache key than identical non-forced requests, so a user-triggered refresh populates a separate cache entry while normal follow-up requests continue serving stale cached data under the old key.
   
   Validate the correctness of the flagged issue. If correct, How can I resolve this? If you propose a fix, implement it and please make it concise.
   If a suggested approach is provided above, use it as the authoritative instruction. If no explicit code suggestion is given, you MUST still draft and apply your own minimal, localized fix — do not punt back with 'no suggestion provided, review manually'. Keep the change as small as possible: add a guard clause, gate on a loading state, reorder an await, wrap in a conditional, etc. Do not refactor surrounding code or expand scope beyond the finding.
   Once fix is implemented, also check other comments on the same PR, and ask user if the user wants to fix the rest of the comments as well. if said yes, then fetch all the comments validate the correctness and implement a minimal fix
   ```
   </details>



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org


Re: [PR] feat(semantic-layers): containment-aware query cache [superset]

Posted by "rusackas (via GitHub)" <gi...@apache.org>.
rusackas closed pull request #40221: feat(semantic-layers): containment-aware query cache
URL: https://github.com/apache/superset/pull/40221


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org


Re: [PR] feat(semantic-layers): containment-aware query cache [superset]

Posted by "codecov[bot] (via GitHub)" <gi...@apache.org>.
codecov[bot] commented on PR #40221:
URL: https://github.com/apache/superset/pull/40221#issuecomment-4478537225

   ## [Codecov](https://app.codecov.io/gh/apache/superset/pull/40221?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
   :x: Patch coverage is `17.02128%` with `429 lines` in your changes missing coverage. Please review.
   :white_check_mark: Project coverage is 55.21%. Comparing base ([`64dae07`](https://app.codecov.io/gh/apache/superset/commit/64dae07675ed840e7a2818b720e1fc4f04b65e43?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`411d040`](https://app.codecov.io/gh/apache/superset/commit/411d0401309e91b491a5f635a3cfca7147164c72?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
   :warning: Report is 63 commits behind head on master.
   
   | [Files with missing lines](https://app.codecov.io/gh/apache/superset/pull/40221?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
   |---|---|---|
   | [superset/semantic\_layers/cache.py](https://app.codecov.io/gh/apache/superset/pull/40221?src=pr&el=tree&filepath=superset%2Fsemantic_layers%2Fcache.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-c3VwZXJzZXQvc2VtYW50aWNfbGF5ZXJzL2NhY2hlLnB5) | 18.00% | [337 Missing :warning: ](https://app.codecov.io/gh/apache/superset/pull/40221?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
   | [superset/semantic\_layers/mapper.py](https://app.codecov.io/gh/apache/superset/pull/40221?src=pr&el=tree&filepath=superset%2Fsemantic_layers%2Fmapper.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-c3VwZXJzZXQvc2VtYW50aWNfbGF5ZXJzL21hcHBlci5weQ==) | 7.14% | [91 Missing :warning: ](https://app.codecov.io/gh/apache/superset/pull/40221?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
   | [superset/common/utils/query\_cache\_manager.py](https://app.codecov.io/gh/apache/superset/pull/40221?src=pr&el=tree&filepath=superset%2Fcommon%2Futils%2Fquery_cache_manager.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-c3VwZXJzZXQvY29tbW9uL3V0aWxzL3F1ZXJ5X2NhY2hlX21hbmFnZXIucHk=) | 66.66% | [1 Missing :warning: ](https://app.codecov.io/gh/apache/superset/pull/40221?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
   > :exclamation:  There is a different number of reports uploaded between BASE (64dae07) and HEAD (411d040). Click for more details.
   > 
   > <details><summary>HEAD has 12 uploads less than BASE</summary>
   >
   >| Flag | BASE (64dae07) | HEAD (411d040) |
   >|------|------|------|
   >|hive|2|1|
   >|python|12|2|
   >|presto|2|1|
   ></details>
   
   <details><summary>Additional details and impacted files</summary>
   
   
   
   ```diff
   @@            Coverage Diff             @@
   ##           master   #40221      +/-   ##
   ==========================================
   - Coverage   64.14%   55.21%   -8.94%     
   ==========================================
     Files        2590     2592       +2     
     Lines      138084   138670     +586     
     Branches    32037    32196     +159     
   ==========================================
   - Hits        88578    76562   -12016     
   - Misses      47985    61445   +13460     
   + Partials     1521      663     -858     
   ```
   
   | [Flag](https://app.codecov.io/gh/apache/superset/pull/40221/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
   |---|---|---|
   | [hive](https://app.codecov.io/gh/apache/superset/pull/40221/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `39.26% <16.24%> (-0.17%)` | :arrow_down: |
   | [mysql](https://app.codecov.io/gh/apache/superset/pull/40221/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
   | [postgres](https://app.codecov.io/gh/apache/superset/pull/40221/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
   | [presto](https://app.codecov.io/gh/apache/superset/pull/40221/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `40.94% <17.02%> (-0.18%)` | :arrow_down: |
   | [python](https://app.codecov.io/gh/apache/superset/pull/40221/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `41.01% <17.02%> (-19.63%)` | :arrow_down: |
   | [sqlite](https://app.codecov.io/gh/apache/superset/pull/40221/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
   | [unit](https://app.codecov.io/gh/apache/superset/pull/40221/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
   
   Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
   </details>
   
   [:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/apache/superset/pull/40221?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).   
   :loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
   <details><summary> :rocket: New features to boost your workflow: </summary>
   
   - :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
   - :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org


Re: [PR] feat(semantic-layers): containment-aware query cache [superset]

Posted by "bito-code-review[bot] (via GitHub)" <gi...@apache.org>.
bito-code-review[bot] commented on code in PR #40221:
URL: https://github.com/apache/superset/pull/40221#discussion_r3272237008


##########
superset-frontend/src/features/semanticLayers/jsonFormsHelpers.tsx:
##########
@@ -255,14 +255,96 @@ const EnumNamesRenderer = withJsonFormsControlProps(EnumNamesControl);
 const enumNamesEntry = {
   // Rank 5: higher than the default string renderer (2–3) so this fires
   // whenever x-enumNames is present, regardless of the underlying type.
+  // Array-of-enum schemas are handled by ``multiEnumEntry`` below — this
+  // renderer only targets scalar string/number controls.
   tester: rankWith(
     5,
+    and(
+      schemaMatches(s => {
+        const names = (s as Record<string, unknown>)['x-enumNames'];
+        return Array.isArray(names) && (names as unknown[]).length > 0;
+      }),
+      schemaMatches(s => (s as Record<string, unknown>)?.type !== 'array'),
+    ),
+  ),
+  renderer: EnumNamesRenderer,
+};
+
+/**
+ * Renderer for ``{type: 'array', items: {enum: [...]}}`` schemas.  Renders
+ * a single Antd Select with ``mode="multiple"`` (tag-style multi-select),
+ * matching the natural expectation of a "pick several from a list" control.
+ *
+ * Without this, the default ``PrimitiveArrayControl`` from the upstream
+ * library renders an "Add …" button that creates one single-select per
+ * element — visually wrong for an enum multi-select and unable to display
+ * ``items.x-enumNames`` labels.
+ *
+ * The renderer is dynamic-aware: when the host form is refreshing the
+ * schema (e.g. compatible options narrowing as the user picks), the Select
+ * shows a loading indicator without becoming disabled, so the user can
+ * continue editing while options refresh.
+ */
+function MultiEnumControl(props: ControlProps) {
+  const { refreshingSchema } = props.config ?? {};
+  const arraySchema = props.schema as Record<string, unknown>;
+  const itemsSchema =
+    (arraySchema.items as Record<string, unknown>) ??
+    ({} as Record<string, unknown>);
+
+  const enumValues = (itemsSchema.enum as unknown[]) ?? [];
+  const enumNames =
+    (itemsSchema['x-enumNames'] as string[]) ?? enumValues.map(String);
+
+  const options = enumValues.map((value, index) => ({
+    value: value as string | number,
+    label: enumNames[index] ?? String(value),
+  }));
+
+  const value = Array.isArray(props.data) ? (props.data as unknown[]) : [];
+
+  const tooltip = (props.uischema?.options as Record<string, unknown>)
+    ?.tooltip as string | undefined;
+
+  return (
+    <Form.Item label={props.label} tooltip={tooltip}>
+      <Select
+        mode="multiple"
+        value={value as (string | number)[]}
+        onChange={next => props.handleChange(props.path, next)}
+        options={options}
+        style={{ width: '100%' }}
+        disabled={!props.enabled}
+        loading={!!refreshingSchema}
+        allowClear
+        optionFilterProp="label"
+        placeholder={
+          (props.uischema?.options as Record<string, unknown>)
+            ?.placeholderText as string | undefined
+        }

Review Comment:
   <div>
   
   
   <div id="suggestion">
   <div id="issue"><b>Missing translation wrapper</b></div>
   <div id="fix">
   
   Wrap the placeholderText from uischema options with the translation function `t()`, for example:
   ```jsx
   placeholder={
     t((props.uischema?.options as Record<string, unknown>)?.placeholderText as string)
   }
   ```
   </div>
   
   
   </div>
   
   
   
   
   <small><i>Code Review Run #983322</i></small>
   </div>
   
   ---
   Should Bito avoid suggestions like this for future reviews? (<a href=https://alpha.bito.ai/home/ai-agents/review-rules>Manage Rules</a>)
   - [ ] Yes, avoid them



##########
superset-frontend/packages/superset-ui-core/src/components/CachedLabel/TooltipContent.tsx:
##########
@@ -23,15 +23,28 @@ import { extendedDayjs } from '../../utils/dates';
 
 interface Props {
   cachedTimestamp?: string;
+  cacheSource?: 'query' | 'semantic';
 }
-export const TooltipContent: FC<Props> = ({ cachedTimestamp }) => {
+export const TooltipContent: FC<Props> = ({
+  cachedTimestamp,
+  cacheSource = 'query',
+}) => {
+  const loadedFromText =
+    cacheSource === 'semantic'
+      ? t('Loaded from semantic smart cache')
+      : t('Loaded data cached');
+  const loadedFallbackText =
+    cacheSource === 'semantic'
+      ? t('Loaded from semantic smart cache')
+      : t('Loaded from cache');

Review Comment:
   <div>
   
   
   <div id="suggestion">
   <div id="issue"><b>Semantic duplication in diff</b></div>
   <div id="fix">
   
   Lines 32-39 introduce semantic duplication: `loadedFromText` and `loadedFallbackText` contain nearly identical conditional logic checking `cacheSource === 'semantic'`. Both must be updated together if cache source logic changes, creating divergence risk. Consider consolidating into a single helper or inline ternary.
   </div>
   
   
   <details>
   <summary>
   <b>Code suggestion</b>
   </summary>
   <blockquote>Check the AI-generated fix before applying</blockquote>
   <div id="code">
   
   
   ```
    --- superset-frontend/packages/superset-ui-core/src/components/CachedLabel/TooltipContent.tsx (lines 32-39) ---
    32: -  const loadedFromText =
    33: -    cacheSource === 'semantic'
    34: -      ? t('Loaded from semantic smart cache')
    35: -      : t('Loaded data cached');
    36: -  const loadedFallbackText =
    37: -    cacheSource === 'semantic'
    38: -      ? t('Loaded from semantic smart cache')
    39: -      : t('Loaded from cache');
    40: +
    41: +  const getLoadedText = (useFallback: boolean) => {
    42: +    if (cacheSource === 'semantic') {
    43: +      return t('Loaded from semantic smart cache');
    44: +    }
    45: +    return useFallback ? t('Loaded from cache') : t('Loaded data cached');
    46: +  };
    47: +
    48:    const cachedText = cachedTimestamp ? (
    49:      <span>
    50: -      {loadedFromText}
    51: +      {getLoadedText(false)}
    52:        <b> {extendedDayjs.utc(cachedTimestamp).fromNow()}</b>
    53:      </span>
    54:    ) : (
    55: -    loadedFallbackText
    56: +    getLoadedText(true)
    57:    );
   ```
   
   </div>
   </details>
   
   
   
   </div>
   
   
   
   
   <small><i>Code Review Run #983322</i></small>
   </div>
   
   ---
   Should Bito avoid suggestions like this for future reviews? (<a href=https://alpha.bito.ai/home/ai-agents/review-rules>Manage Rules</a>)
   - [ ] Yes, avoid them



##########
superset-frontend/packages/superset-ui-core/src/components/CachedLabel/TooltipContent.test.tsx:
##########
@@ -36,3 +36,14 @@ test('Rendering TooltipContent correctly - with timestep', () => {
       .fromNow()}. Click to force-refresh`,
   );
 });
+
+test('Rendering TooltipContent correctly - semantic cache', () => {
+  render(
+    <TooltipContent cacheSource="semantic" cachedTimestamp="01-01-2000" />,
+  );
+  expect(screen.getByTestId('tooltip-content')?.textContent).toBe(

Review Comment:
   <div>
   
   
   <div id="suggestion">
   <div id="issue"><b>data-testid attribute mismatch</b></div>
   <div id="fix">
   
   The component uses `data-test="tooltip-content"` but `getByTestId` from React Testing Library searches for `data-testid`. This mismatch causes the test to fail at runtime — the query returns `undefined` instead of finding the element. All three test cases (lines 26, 33, 44) use this pattern and will fail.
   </div>
   
   
   <details>
   <summary>
   <b>Code suggestion</b>
   </summary>
   <blockquote>Check the AI-generated fix before applying</blockquote>
   <div id="code">
   
   
   ```
    --- superset-frontend/packages/superset-ui-core/src/components/CachedLabel/TooltipContent.tsx
    +++ superset-frontend/packages/superset-ui-core/src/components/CachedLabel/TooltipContent.tsx
    @@ -48,7 +48,7 @@ export const TooltipContent: FC<Props> = ({
       );
    
       return (
    -    <span data-test="tooltip-content">
    +    <span data-testid="tooltip-content">
           {cachedText}. {t('Click to force-refresh')}
         </span>
       );
     };
   ```
   
   </div>
   </details>
   
   
   
   </div>
   
   
   
   
   <small><i>Code Review Run #983322</i></small>
   </div>
   
   ---
   Should Bito avoid suggestions like this for future reviews? (<a href=https://alpha.bito.ai/home/ai-agents/review-rules>Manage Rules</a>)
   - [ ] Yes, avoid them



##########
superset/semantic_layers/cache.py:
##########
@@ -0,0 +1,759 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""
+Containment-aware cache for semantic view queries.
+
+A broader cached result can satisfy a narrower new query: when the new query's
+filters and limit are strictly more restrictive than a cached entry's, the cached
+DataFrame is post-filtered and re-limited rather than re-executing the underlying
+query.
+
+See ``docs/`` and the plan file for the design rationale; the rules summary:
+
+* Same metrics and dimensions (shape).
+* Each cached filter must be implied by a new-query filter on the same column.
+* New filters on columns with no cached constraint are applied post-fetch as
+  "leftovers" — provided the column is in the projection.
+* Cached ``limit`` must be at least the new ``limit``; if a cached ``limit`` is
+  present, the orderings must match (otherwise the cached "top N" is not the
+  true top of the new query).
+* ``ADHOC`` and ``HAVING`` filters require exact-set equality.
+* ``offset != 0`` and mismatching ``group_limit`` skip the cache.
+"""
+
+from __future__ import annotations
+
+import logging
+import re
+import time as _time
+from dataclasses import dataclass, field
+from datetime import date, datetime, time, timedelta
+from enum import Enum
+from typing import Any, Iterable
+
+import pandas as pd
+import pyarrow as pa
+from flask import current_app
+from superset_core.semantic_layers.types import (
+    AdhocExpression,
+    AggregationType,
+    Dimension,
+    Filter,
+    Metric,
+    Operator,
+    OrderDirection,
+    OrderTuple,
+    PredicateType,
+    SemanticQuery,
+    SemanticRequest,
+    SemanticResult,
+)
+
+from superset.extensions import cache_manager
+from superset.utils import json
+from superset.utils.hashing import hash_from_str
+from superset.utils.pandas_postprocessing.aggregate import aggregate
+
+logger = logging.getLogger(__name__)
+
+INDEX_KEY_PREFIX = "sv:idx:"
+VALUE_KEY_PREFIX = "sv:val:"
+MAX_ENTRIES_PER_VIEW = 128
+
+_AGGREGATION_TO_PANDAS: dict[AggregationType, str] = {
+    AggregationType.SUM: "sum",
+    AggregationType.COUNT: "sum",
+    AggregationType.MIN: "min",
+    AggregationType.MAX: "max",
+}
+ADDITIVE_AGGREGATIONS = frozenset(_AGGREGATION_TO_PANDAS)
+
+
+@dataclass(frozen=True)
+class ViewMeta:
+    """Identity/freshness/TTL info pulled from the SemanticView ORM row."""
+
+    uuid: str
+    changed_on_iso: str
+    cache_timeout: int | None
+
+
+class ReuseMode(Enum):
+    """How a cached entry maps onto the new query."""
+
+    EXACT = "exact"  # same dims and metrics; only leftovers/limit/order to apply
+    PROJECT = "project"  # same dims, cached metrics ⊃ new metrics; drop columns
+    ROLLUP = "rollup"  # cached dims ⊃ new dims; re-aggregate
+
+
+# Preference order when multiple cached entries satisfy a query: pick the one
+# requiring the least post-processing.
+_MODE_RANK: dict[ReuseMode, int] = {
+    ReuseMode.EXACT: 0,
+    ReuseMode.PROJECT: 1,
+    ReuseMode.ROLLUP: 2,
+}
+
+
+@dataclass(frozen=True)
+class CachedEntry:
+    filters: frozenset[Filter]
+    dimension_keys: frozenset[str]
+    metric_ids: frozenset[str]
+    limit: int | None
+    offset: int
+    order_key: str
+    group_limit_key: str
+    value_key: str
+    timestamp: float = field(default_factory=_time.time)
+
+
+# ---------------------------------------------------------------------------
+# Public surface
+# ---------------------------------------------------------------------------
+
+
+def try_serve_from_cache(
+    view_meta: ViewMeta,
+    query: SemanticQuery,
+) -> SemanticResult | None:
+    """Return a cached ``SemanticResult`` that satisfies ``query`` if any."""
+    try:
+        cache = cache_manager.data_cache
+        idx_key = shape_key(view_meta, query)
+        entries: list[CachedEntry] | None = cache.get(idx_key)
+        if not entries:
+            return None
+
+        # Rank candidates so EXACT is preferred over PROJECT over ROLLUP. With
+        # one bucket per view, multiple entries may satisfy a query; pick the
+        # one needing the least post-processing.
+        candidates: list[tuple[int, CachedEntry, set[Filter], ReuseMode]] = []
+        for entry in entries:
+            ok, leftovers, mode = can_satisfy(entry, query)
+            if ok:
+                candidates.append((_MODE_RANK[mode], entry, leftovers, mode))
+        candidates.sort(key=lambda c: c[0])
+
+        served: SemanticResult | None = None
+        for _, entry, leftovers, mode in candidates:
+            payload = cache.get(entry.value_key)
+            if payload is None:
+                continue
+            if mode == ReuseMode.ROLLUP and not _projection_input_complete(
+                entry, payload
+            ):
+                continue
+            served = _apply_post_processing(payload, query, leftovers, mode)
+            break
+
+        # Drop index entries whose values have been evicted.
+        pruned = [e for e in entries if cache.get(e.value_key) is not None]
+        if len(pruned) != len(entries):
+            cache.set(idx_key, pruned, timeout=_timeout(view_meta))
+        return served
+    except Exception:  # pragma: no cover - defensive
+        logger.warning("Semantic view cache lookup failed", exc_info=True)
+        return None
+
+
+def store_result(
+    view_meta: ViewMeta,
+    query: SemanticQuery,
+    result: SemanticResult,
+) -> None:
+    """Persist ``result`` under a fresh value key and register a descriptor."""
+    try:
+        cache = cache_manager.data_cache
+        timeout = _timeout(view_meta)
+        vkey = value_key(view_meta, query)
+        cache.set(vkey, result, timeout=timeout)
+
+        idx_key = shape_key(view_meta, query)
+        entries: list[CachedEntry] = list(cache.get(idx_key) or [])
+        entry = CachedEntry(
+            filters=frozenset(query.filters or set()),
+            dimension_keys=frozenset(_dimension_key(d) for d in query.dimensions),
+            metric_ids=frozenset(m.id for m in query.metrics),
+            limit=query.limit,
+            offset=query.offset or 0,
+            order_key=_order_key(query.order),
+            group_limit_key=_group_limit_key(query.group_limit),
+            value_key=vkey,
+        )
+        entries = [e for e in entries if e.value_key != vkey]
+        entries.append(entry)
+        if len(entries) > MAX_ENTRIES_PER_VIEW:
+            entries = sorted(entries, key=lambda e: e.timestamp)[-MAX_ENTRIES_PER_VIEW:]
+        cache.set(idx_key, entries, timeout=timeout)
+    except Exception:  # pragma: no cover - defensive

Review Comment:
   <div>
   
   
   <div id="suggestion">
   <div id="issue"><b>Avoid catching blind exception</b></div>
   <div id="fix">
   
   Replace the broad `Exception` catch with specific exception types to improve error handling. Similar issue exists at line 170.
   </div>
   
   
   <details>
   <summary>
   <b>Code suggestion</b>
   </summary>
   <blockquote>Check the AI-generated fix before applying</blockquote>
   <div id="code">
   
   
   ````suggestion
       except (KeyError, ValueError, TypeError):  # pragma: no cover - defensive
   ````
   
   </div>
   </details>
   
   
   
   </div>
   
   
   
   
   <small><i>Code Review Run #983322</i></small>
   </div>
   
   ---
   Should Bito avoid suggestions like this for future reviews? (<a href=https://alpha.bito.ai/home/ai-agents/review-rules>Manage Rules</a>)
   - [ ] Yes, avoid them



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org


Re: [PR] feat(semantic-layers): containment-aware query cache [superset]

Posted by "mikebridge (via GitHub)" <gi...@apache.org>.
mikebridge commented on PR #40221:
URL: https://github.com/apache/superset/pull/40221#issuecomment-5480724838

   Closing: superseded by #42760 (consolidated semantic-cache), per the supersession notes posted across this stack on 08-06. The rest of the stack (#41824/#41825/#41826/#41856) was closed 08-28; this one was missed.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org


Re: [PR] feat(semantic-layers): containment-aware query cache [superset]

Posted by "mikebridge (via GitHub)" <gi...@apache.org>.
mikebridge commented on PR #40221:
URL: https://github.com/apache/superset/pull/40221#issuecomment-4898056139

   Follow-up to the triage above — the review panel on #41825 surfaced two more findings in this branch, both now resolved or characterized:
   
   1. **Forced refresh never replaced the stale *semantic*-cache entry** (verified): `_make_cached_dispatch` returned the raw dispatcher on `force_query`, bypassing both the read and `store_result` — so other requests' containment lookups kept serving the stale entry until TTL. Fixed in **#41826** (into this branch): forced queries now skip only the read and store their fresh result, with the force test rewritten to pin the full contract (validated to fail against the pre-fix mapper). Together with #41825 this makes force-refresh fully effective across both cache layers.
   2. **`force_query` also enters `cache_values`** via the contribution-totals merge (`query_context_processor.py:346`, `{**cached_query, **query.to_dict()}`). Verified consequence is narrower than it first looks: nothing rebuilds a QueryContext from `cache_values` (no permanent-bypass path), but `QueryContext.cache_key()` (the `qc-` payload key) hashes `cache_values` — so on contribution charts, a forced request's payload caches under a divergent `qc-` key. Same bug family, low impact; the clean fix would strip execution-control keys from that merge or in `QueryContext.cache_key()`. Left for this branch's owner to place — happy to do a fourth micro-PR if wanted.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org


Re: [PR] feat(semantic-layers): containment-aware query cache [superset]

Posted by "rusackas (via GitHub)" <gi...@apache.org>.
rusackas commented on PR #40221:
URL: https://github.com/apache/superset/pull/40221#issuecomment-5579727669

   Looks like this one slipped through when the rest of the stack closed out on 08-28. It's superseded by #42760, so closing now. Holler if that's off and you think we should reopen.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org


Re: [PR] feat(semantic-layers): containment-aware query cache [superset]

Posted by "bito-code-review[bot] (via GitHub)" <gi...@apache.org>.
bito-code-review[bot] commented on PR #40221:
URL: https://github.com/apache/superset/pull/40221#issuecomment-4478328840

   AI Code Review is in progress (usually takes 3 to 15 minutes unless it's a very large PR).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org


Re: [PR] feat(semantic-layers): containment-aware query cache [superset]

Posted by "mikebridge (via GitHub)" <gi...@apache.org>.
mikebridge commented on PR #40221:
URL: https://github.com/apache/superset/pull/40221#issuecomment-4898567658

   Two more base-branch observations from the #41826 review panel, for this branch's backlog (neither introduced by the stacked PRs, neither urgent):
   
   1. **Forward-looking cache-key design note**: the semantic cache's keys carry no user/RLS/impersonation context — correct today because semantic-view execution is user-agnostic, but it means *every* store path becomes a cross-user leak if semantic-layer engines ever gain per-user execution context (engine impersonation, per-user RLS in the underlying semantic platform). Core Superset handles this by folding RLS ids into cache keys. Worth a comment in `cache.py` documenting the user-agnostic assumption, so the constraint is visible when per-user execution is considered.
   2. **Provenance gap on the cheapest hit path**: `_apply_post_processing` early-returns the cached payload as-is for EXACT-mode / no-leftovers / no-limit hits, skipping the `SemanticRequest(type="cache")` note — so `is_cached` reads `False` for exactly the most common cache hit, and the Explore "semantic smart cache" indicator won't show. One-line fix (append the provenance record before the early return).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org


Re: [PR] feat(semantic-layers): containment-aware query cache [superset]

Posted by "mikebridge (via GitHub)" <gi...@apache.org>.
mikebridge commented on code in PR #40221:
URL: https://github.com/apache/superset/pull/40221#discussion_r3532281674


##########
superset/common/query_object.py:
##########
@@ -128,6 +129,7 @@ def __init__(  # pylint: disable=too-many-locals, too-many-arguments
         post_processing: list[dict[str, Any] | None] | None = None,
         row_limit: int | None = None,
         row_offset: int | None = None,
+        force_query: bool = False,
         series_columns: list[Column] | None = None,

Review Comment:
   Verified real — `cache_key()` strips `from_dttm`/`to_dttm` but not `force_query`, so a forced refresh writes under a different key while ordinary requests keep reading the stale entry; affects every chart on the legacy path. Fix (one line + a key-equality regression test, validated to fail without it) opened into this branch as **#41825**.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org


Re: [PR] feat(semantic-layers): containment-aware query cache [superset]

Posted by "rusackas (via GitHub)" <gi...@apache.org>.
rusackas commented on PR #40221:
URL: https://github.com/apache/superset/pull/40221#issuecomment-4488505624

   Dropping some "findings" here in case they're helpful:
   
   1. Thread safety / race condition in store_result
   The store operation involves: (1) read the index, (2) append the new entry, (3) write the index back. This is a read-modify-write on a Redis key that is not atomic. Under concurrent writes for the same view, two workers could read the same index simultaneously, each append their entry, and one write would overwrite the other's addition. A Redis WATCH/MULTI/EXEC transaction or a Lua script would eliminate this race. Under normal Superset load with one worker per view, this is unlikely but not impossible — especially in a heavily concurrent dashboard load.
   
   2. Test coverage is critically low: 19.93% patch coverage
   
   3. The field import in from dataclasses import dataclass, field (line 44) — field doesn't appear to be used in CachedEntry. If unused, it should be removed.
   
   4. The _sql_like_to_regex function is listed in the symbol table — this is a non-trivial implementation. LIKE pattern → regex conversion has many edge cases (%, _, escaping). Make sure the test suite covers escaped wildcard characters (e.g., LIKE '100\%'
   
   5. ROLLUP correctness: guarding against stale cache entries
   When the cache tries to serve a ROLLUP result, there are two reasons it might skip a candidate and move to the next one:
   
   * The value was evicted from Redis — Redis may have deleted the cached data (due to TTL or memory pressure) between when the code found it in the index and when it tried to fetch it. If that happens, the code correctly skips it.
   * The cached result is incomplete — If a cached query had a row limit and that limit was fully reached, we can't safely roll up those rows to answer a different grouping (because we're only seeing the "top N" rows of the original query, not the full dataset). So the code also skips those.
   
   The concern is that these two checks happen in sequence, and the interaction between them should be well-tested. Specifically: what if an entry is evicted and would have been rejected as incomplete anyway? What if the only ROLLUP candidate gets evicted and there's no fallback? The good news is that `test_serve_falls_back_to_rollup_when_exact_value_evicted` in the integration test does cover the eviction fallback path, which addresses the main worry. This is more of a "make sure the test coverage keeps up with the edge cases" note than a definitive bug.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org


Re: [PR] feat(semantic-layers): containment-aware query cache [superset]

Posted by "bito-code-review[bot] (via GitHub)" <gi...@apache.org>.
bito-code-review[bot] commented on PR #40221:
URL: https://github.com/apache/superset/pull/40221#issuecomment-4495409359

   AI Code Review is in progress (usually takes 3 to 15 minutes unless it's a very large PR).
   
   <details>
   <summary>
   Bito Usage Guide
   </summary>
   
   **Commands**
         
   Type the following command in the pull request comment and save the comment.
         
   - `/review` - Manually triggers a full AI review.
   
   - `/pause` - Pauses automatic reviews on this pull request.
   - `/resume` - Resumes automatic reviews.
   - `/resolve` - Marks all Bito-posted review comments as resolved.
   - `/abort` - Cancels all in-progress reviews.
   
         
   Refer to the <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/available-commands" target="_blank" rel="noopener noreferrer">documentation</a> for additional commands.
         
   **Configuration**
   
   This repository uses `Superset` You can customize the agent settings <a href="https://alpha.bito.ai/home/ai-agents/code-review-agent" target="_blank" rel="noopener noreferrer">here</a> or contact your Bito workspace admin at evan@preset.io.
         
   **Documentation & Help**
   - <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/getting-started/install-run-using-bito-cloud/create-or-customize-an-agent-instance" target="_blank" rel="noopener noreferrer">Customize agent settings</a>
   - <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/implementing-custom-code-review-rules" target="_blank" rel="noopener noreferrer">Review rules</a>
   - <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent" target="_blank" rel="noopener noreferrer">General documentation</a>
   - <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/faqs" target="_blank" rel="noopener noreferrer">FAQ</a>
   
   </details>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org


Re: [PR] feat(semantic-layers): containment-aware query cache [superset]

Posted by "mikebridge (via GitHub)" <gi...@apache.org>.
mikebridge commented on PR #40221:
URL: https://github.com/apache/superset/pull/40221#issuecomment-5206522918

   Closing to prevent accidental merge: #42760 is the consolidated replacement under review and explicitly carries forward the still-valid work and attribution from this PR stack. This does not mean #42760 has been accepted for adoption yet; that decision remains pending. If the consolidation path changes, this PR can be reopened.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org


Re: [PR] feat(semantic-layers): containment-aware query cache [superset]

Posted by "mikebridge (via GitHub)" <gi...@apache.org>.
mikebridge commented on PR #40221:
URL: https://github.com/apache/superset/pull/40221#issuecomment-4897400012

   @betodealmeida — I've opened #41824 **into this branch** addressing @rusackas's `store_result` race finding (twice-flagged above): the index read-modify-write (and the eviction prune, which has the same shape) now go through a short cache-level lock via `cache.add`, with the invariant that the index is only written while holding it. Notes on why the suggested WATCH/Lua route doesn't fit (pickled payloads + the Flask-Caching abstraction) are in that PR, along with a **deterministic two-thread regression test** that reproduces the clobber on the current code every run and passes with the fix. Feel free to merge it into `sl-cache`, cherry-pick, or tell me what you'd like different.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org