You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@superset.apache.org by "rusackas (via GitHub)" <gi...@apache.org> on 2026/06/18 06:09:17 UTC
[PR] [SIP] Correct totals/subtotals for non-additive metrics (Table + Pivot) [superset]
rusackas opened a new pull request, #41184:
URL: https://github.com/apache/superset/pull/41184
### SUMMARY
Draft SIP + **prototype development zone** for fixing one of Superset's
longest-standing correctness gaps: **totals and subtotals are wrong for
non-additive metrics** (ratios, `COUNT_DISTINCT`, `AVG`, percentiles, and
"Percentage metrics") in both the Table and Pivot Table charts.
The proposal lives in [`SIP.md`](SIP.md). This PR is intentionally a draft so
the SIP, a failing-test matrix of the known reported cases, and the POC
implementation can be rounded out together (TDD: the POC drives the matrix
green).
**Core principle:** totals must be computed by the DB at the grouping
granularity, never by re-aggregating already-aggregated cells. A SQL aggregate
metric is correct at any grouping level if the same expression is evaluated
grouped at that level, so no formula language is needed for the common cases.
**Approach:** detect additivity (additive-only = current cheap path, unchanged)
→ for non-additive, compute rollup levels via native `GROUPING SETS` in a
single query → fall back to multi-query (the #34592 approach) on engines lacking
a new `supports_grouping_sets` capability → recompute (don't sum) post-processing
% columns → one shared mechanism for Table + Pivot.
Consolidates: #25747 (canonical), #32260, #38674, #36165, #37627,
#34350/#34425/#34426. Design thread: #29297. Builds on prior art in #34592.
Out of scope: #39223, SIP-179 (#34245), #35089, SIP-205 (#38586).
### BEFORE/AFTER SCREENSHOTS OR ANIMATED GIF
N/A (proposal/scaffold).
### TESTING INSTRUCTIONS
The TDD acceptance matrix is in `SIP.md` (§ Validation / TDD test matrix).
Failing tests and POC to follow in this PR.
### ADDITIONAL INFORMATION
- [ ] Has associated issue: relates to #25747, #29297
- [x] Required feature flags: TBD (possible opt-in flag during rollout)
- [ ] Changes UI
- [ ] Includes DB Migration (please mention transaction, lock, and revert plan)
- [ ] Introduces new feature or API
- [ ] Removes existing feature or API
> Draft — not for review yet. Opening early as a shared SIP + prototype space.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] [SIP] Correct totals/subtotals for non-additive metrics (Table + Pivot) [superset]
Posted by "codecov[bot] (via GitHub)" <gi...@apache.org>.
codecov[bot] commented on PR #41184:
URL: https://github.com/apache/superset/pull/41184#issuecomment-4744230915
## [Codecov](https://app.codecov.io/gh/apache/superset/pull/41184?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:white_check_mark: All modified and coverable lines are covered by tests.
:white_check_mark: Project coverage is 54.84%. Comparing base ([`ae0b1f0`](https://app.codecov.io/gh/apache/superset/commit/ae0b1f0308af08ea706bca196c40ba186b2c3daa?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`b685383`](https://app.codecov.io/gh/apache/superset/commit/b685383445f6878998a08a9eb80fc4dc76881bdd?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
:warning: Report is 11 commits behind head on master.
> :exclamation: There is a different number of reports uploaded between BASE (ae0b1f0) and HEAD (b685383). Click for more details.
>
> <details><summary>HEAD has 40 uploads less than BASE</summary>
>
>| Flag | BASE (ae0b1f0) | HEAD (b685383) |
>|------|------|------|
>|python|36|1|
>|hive|6|1|
></details>
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #41184 +/- ##
==========================================
- Coverage 64.31% 54.84% -9.48%
==========================================
Files 2651 2651
Lines 144773 144773
Branches 33407 33407
==========================================
- Hits 93111 79394 -13717
- Misses 49992 64980 +14988
+ Partials 1670 399 -1271
```
| [Flag](https://app.codecov.io/gh/apache/superset/pull/41184/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [hive](https://app.codecov.io/gh/apache/superset/pull/41184/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `39.32% <ø> (ø)` | |
| [mysql](https://app.codecov.io/gh/apache/superset/pull/41184/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [postgres](https://app.codecov.io/gh/apache/superset/pull/41184/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [presto](https://app.codecov.io/gh/apache/superset/pull/41184/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [python](https://app.codecov.io/gh/apache/superset/pull/41184/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `39.32% <ø> (-20.25%)` | :arrow_down: |
| [sqlite](https://app.codecov.io/gh/apache/superset/pull/41184/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [unit](https://app.codecov.io/gh/apache/superset/pull/41184/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
</details>
[:umbrella: View full report in Codecov by Harness](https://app.codecov.io/gh/apache/superset/pull/41184?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
- :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(table/pivot-table): correct non-additive totals/subtotals via DB rollup [POC for SIP-216] [superset]
Posted by "rusackas (via GitHub)" <gi...@apache.org>.
rusackas commented on code in PR #41184:
URL: https://github.com/apache/superset/pull/41184#discussion_r3564072744
##########
superset/db_engine_specs/presto.py:
##########
@@ -166,6 +166,7 @@ class PrestoBaseEngineSpec(BaseEngineSpec, metaclass=ABCMeta):
supports_dynamic_schema = True
supports_catalog = supports_dynamic_catalog = supports_cross_catalog_queries = True
+ supports_grouping_sets = True
Review Comment:
Fair point — moved `supports_grouping_sets = True` off the shared `PrestoBaseEngineSpec` down to `PrestoEngineSpec`/`TrinoEngineSpec` directly, and added an explicit `= False` override on `HiveEngineSpec` since it (and Spark/Databricks-Hive by inheritance) extend `PrestoEngineSpec` and would otherwise still inherit the flag.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(table/pivot-table): correct non-additive totals/subtotals via DB rollup [POC for SIP-216] [superset]
Posted by "bito-code-review[bot] (via GitHub)" <gi...@apache.org>.
bito-code-review[bot] commented on PR #41184:
URL: https://github.com/apache/superset/pull/41184#issuecomment-4947060024
AI Code Review is in progress (usually takes 3 to 15 minutes unless it's a very large PR).
<details>
<summary>
Bito Usage Guide
</summary>
**Commands**
Type the following command in the pull request comment and save the comment.
- `/review` - Manually triggers a full AI review.
- `/pause` - Pauses automatic reviews on this pull request.
- `/resume` - Resumes automatic reviews.
- `/resolve` - Marks all Bito-posted review comments as resolved.
- `/abort` - Cancels all in-progress reviews.
Refer to the <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/available-commands" target="_blank" rel="noopener noreferrer">documentation</a> for additional commands.
**Configuration**
This repository uses `Superset` You can customize the agent settings <a href="https://alpha.bito.ai/home/ai-agents/code-review-agent" target="_blank" rel="noopener noreferrer">here</a> or contact your Bito workspace admin at evan@preset.io.
**Documentation & Help**
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/getting-started/install-run-using-bito-cloud/create-or-customize-an-agent-instance" target="_blank" rel="noopener noreferrer">Customize agent settings</a>
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/implementing-custom-code-review-rules" target="_blank" rel="noopener noreferrer">Review rules</a>
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent" target="_blank" rel="noopener noreferrer">General documentation</a>
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/faqs" target="_blank" rel="noopener noreferrer">FAQ</a>
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(table/pivot-table): correct non-additive totals/subtotals via DB rollup [POC for SIP-216] [superset]
Posted by "codeant-ai-for-open-source[bot] (via GitHub)" <gi...@apache.org>.
codeant-ai-for-open-source[bot] commented on code in PR #41184:
URL: https://github.com/apache/superset/pull/41184#discussion_r3565613580
##########
superset/common/grouping_sets.py:
##########
@@ -0,0 +1,117 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""
+SQL building blocks for the pivot-table non-additive totals optimization
+(SIP.md, phase 3b). When a datasource engine reports
+``supports_grouping_sets``, the N per-rollup-level queries can be collapsed into
+a single ``GROUPING SETS`` query: the database computes every level in one scan,
+and each returned row is attributed to its level via ``GROUPING()`` markers.
+
+These are the engine-agnostic SQL primitives. Wiring them into the query context
+(emitting one query and splitting the result back into per-level results) is the
+remaining integration; see SIP.md.
+"""
+
+from __future__ import annotations
+
+from collections.abc import Sequence
+from typing import Final
+
+import pandas as pd
+from sqlalchemy import func, tuple_
+from sqlalchemy.sql.elements import ColumnElement
+
+# Suffix for the per-column GROUPING() marker columns added to a GROUPING SETS
+# query. Chosen to be unlikely to collide with a real metric/column label.
+GROUPING_MARKER_SUFFIX: Final = "__superset_grouping"
+
+
+def grouping_marker_label(column_label: str) -> str:
+ """The output label of the GROUPING() marker for a groupby column."""
+ return f"{column_label}{GROUPING_MARKER_SUFFIX}"
+
+
+def grouping_sets_clause(
+ groups: Sequence[Sequence[ColumnElement]],
+) -> ColumnElement:
+ """
+ Build a ``GROUP BY GROUPING SETS (...)`` clause from rollup column groups.
+
+ Each group is the set of columns grouped at one rollup level; the empty
+ group ``()`` is the grand total. For example, groups ``[[a, b], [a], []]``
+ produce ``GROUPING SETS ((a, b), (a), ())``.
+
+ :param groups: one column list per rollup level
+ :return: a clause element suitable for ``select(...).group_by(...)``
+ """
+ return func.grouping_sets(*[tuple_(*group) for group in groups])
+
+
+def grouping_id_column(column: ColumnElement, label: str) -> ColumnElement:
+ """
+ Build a ``GROUPING(col) AS label`` marker column.
+
+ In a ``GROUPING SETS`` result, ``GROUPING(col)`` is ``0`` when ``col`` is
+ part of the row's grouping level and ``1`` when it has been rolled up
+ (aggregated away). Selecting one marker per groupby column lets the caller
+ attribute each returned row to its rollup level when splitting the single
+ result back into per-level results.
+
+ :param column: the groupby column to probe
+ :param label: the output label for the marker (see ``grouping_marker_label``)
+ :return: the labelled ``GROUPING(col)`` column
+ """
+ return func.grouping(column).label(label)
+
+
+def split_grouping_sets_result(
+ df: pd.DataFrame,
+ levels: Sequence[Sequence[str]],
+ groupby_columns: Sequence[str],
+) -> list[pd.DataFrame]:
+ """
+ Split a combined ``GROUPING SETS`` result into one DataFrame per rollup
+ level, the inverse of {@link grouping_sets_clause}.
+
+ Each row of ``df`` carries a ``GROUPING()`` marker per groupby column (named
+ by ``grouping_marker_label``): ``0`` if the column is grouped at that row's
+ level, ``1`` if it was rolled up. A row belongs to a level iff its markers
+ are ``0`` exactly for that level's columns. Marker columns are dropped from
+ the returned frames so each looks like an ordinary per-level query result.
+
+ :param df: the combined query result, including marker columns
+ :param levels: the grouped-column list for each rollup level (same order as
+ passed to ``grouping_sets_clause``)
+ :param groupby_columns: every groupby column that has a marker
+ :return: one DataFrame per level, in ``levels`` order
+ """
+ markers: list[str] = [grouping_marker_label(col) for col in groupby_columns]
+ results: list[pd.DataFrame] = []
+ for level in levels:
+ grouped: set[str] = set(level)
+ mask: pd.Series = pd.Series(True, index=df.index)
+ for col in groupby_columns:
+ expected = 0 if col in grouped else 1
Review Comment:
**Suggestion:** Add an explicit type annotation for this local variable to satisfy the type-hint requirement for annotatable variables. [custom_rule]
**Severity Level:** Minor 🧹
<details>
<summary><b>Why it matters? ⭐ </b></summary>
This new local variable has a stable, obvious type (`int`) and is introduced without an annotation. The stated rule requires type hints for annotatable variables, so this is a real violation.
</details>
<details>
<summary><b>Rule source 📖 </b></summary>
.cursor/rules/dev-standard.mdc (line 28)
</details>
[](https://app.codeant.ai/fix-in-ide?tool=cursor&prompt_id=29ba2337be984850a056b054700fcf35&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset) [](https://app.codeant.ai/fix-in-ide?tool=vscode-claude&prompt_id=29ba2337be984850a056b054700fcf35&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset)
*(Use Cmd/Ctrl + Click for best experience)*
<details>
<summary><b>Prompt for AI Agent 🤖 </b></summary>
```mdx
This is a comment left during a code review.
**Path:** superset/common/grouping_sets.py
**Line:** 109:109
**Comment:**
*Custom Rule: Add an explicit type annotation for this local variable to satisfy the type-hint requirement for annotatable variables.
Validate the correctness of the flagged issue. If correct, How can I resolve this? If you propose a fix, implement it and please make it concise.
Once fix is implemented, also check other comments on the same PR, and ask user if the user wants to fix the rest of the comments as well. if said yes, then fetch all the comments validate the correctness and implement a minimal fix
```
</details>
<a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F41184&comment_hash=82120bfd60002996bfa32b74b394c6ac43ef73ed35e5d790e03f109a508dfa7e&reaction=like'>👍</a> | <a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F41184&comment_hash=82120bfd60002996bfa32b74b394c6ac43ef73ed35e5d790e03f109a508dfa7e&reaction=dislike'>👎</a>
##########
superset/common/grouping_sets.py:
##########
@@ -0,0 +1,117 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""
+SQL building blocks for the pivot-table non-additive totals optimization
+(SIP.md, phase 3b). When a datasource engine reports
+``supports_grouping_sets``, the N per-rollup-level queries can be collapsed into
+a single ``GROUPING SETS`` query: the database computes every level in one scan,
+and each returned row is attributed to its level via ``GROUPING()`` markers.
+
+These are the engine-agnostic SQL primitives. Wiring them into the query context
+(emitting one query and splitting the result back into per-level results) is the
+remaining integration; see SIP.md.
+"""
+
+from __future__ import annotations
+
+from collections.abc import Sequence
+from typing import Final
+
+import pandas as pd
+from sqlalchemy import func, tuple_
+from sqlalchemy.sql.elements import ColumnElement
+
+# Suffix for the per-column GROUPING() marker columns added to a GROUPING SETS
+# query. Chosen to be unlikely to collide with a real metric/column label.
+GROUPING_MARKER_SUFFIX: Final = "__superset_grouping"
+
+
+def grouping_marker_label(column_label: str) -> str:
+ """The output label of the GROUPING() marker for a groupby column."""
+ return f"{column_label}{GROUPING_MARKER_SUFFIX}"
+
+
+def grouping_sets_clause(
+ groups: Sequence[Sequence[ColumnElement]],
+) -> ColumnElement:
+ """
+ Build a ``GROUP BY GROUPING SETS (...)`` clause from rollup column groups.
+
+ Each group is the set of columns grouped at one rollup level; the empty
+ group ``()`` is the grand total. For example, groups ``[[a, b], [a], []]``
+ produce ``GROUPING SETS ((a, b), (a), ())``.
+
+ :param groups: one column list per rollup level
+ :return: a clause element suitable for ``select(...).group_by(...)``
+ """
+ return func.grouping_sets(*[tuple_(*group) for group in groups])
+
+
+def grouping_id_column(column: ColumnElement, label: str) -> ColumnElement:
+ """
+ Build a ``GROUPING(col) AS label`` marker column.
+
+ In a ``GROUPING SETS`` result, ``GROUPING(col)`` is ``0`` when ``col`` is
+ part of the row's grouping level and ``1`` when it has been rolled up
+ (aggregated away). Selecting one marker per groupby column lets the caller
+ attribute each returned row to its rollup level when splitting the single
+ result back into per-level results.
+
+ :param column: the groupby column to probe
+ :param label: the output label for the marker (see ``grouping_marker_label``)
+ :return: the labelled ``GROUPING(col)`` column
+ """
+ return func.grouping(column).label(label)
+
+
+def split_grouping_sets_result(
+ df: pd.DataFrame,
+ levels: Sequence[Sequence[str]],
+ groupby_columns: Sequence[str],
+) -> list[pd.DataFrame]:
+ """
+ Split a combined ``GROUPING SETS`` result into one DataFrame per rollup
+ level, the inverse of {@link grouping_sets_clause}.
+
+ Each row of ``df`` carries a ``GROUPING()`` marker per groupby column (named
+ by ``grouping_marker_label``): ``0`` if the column is grouped at that row's
+ level, ``1`` if it was rolled up. A row belongs to a level iff its markers
+ are ``0`` exactly for that level's columns. Marker columns are dropped from
+ the returned frames so each looks like an ordinary per-level query result.
+
+ :param df: the combined query result, including marker columns
+ :param levels: the grouped-column list for each rollup level (same order as
+ passed to ``grouping_sets_clause``)
+ :param groupby_columns: every groupby column that has a marker
+ :return: one DataFrame per level, in ``levels`` order
+ """
+ markers: list[str] = [grouping_marker_label(col) for col in groupby_columns]
+ results: list[pd.DataFrame] = []
+ for level in levels:
+ grouped: set[str] = set(level)
+ mask: pd.Series = pd.Series(True, index=df.index)
+ for col in groupby_columns:
+ expected = 0 if col in grouped else 1
+ mask &= df[grouping_marker_label(col)] == expected
+ level_df = (
+ df[mask]
+ .drop(columns=[m for m in markers if m in df.columns])
+ .reset_index(drop=True)
+ )
Review Comment:
**Suggestion:** Add a concrete type annotation to this local DataFrame variable assignment to keep local variable typing explicit. [custom_rule]
**Severity Level:** Minor 🧹
<details>
<summary><b>Why it matters? ⭐ </b></summary>
The variable clearly holds a `pandas.DataFrame`, and it is introduced without a type annotation. That matches the custom requirement to annotate relevant variables that can be typed.
</details>
<details>
<summary><b>Rule source 📖 </b></summary>
.cursor/rules/dev-standard.mdc (line 28)
</details>
[](https://app.codeant.ai/fix-in-ide?tool=cursor&prompt_id=daf62a6903c04a0abf0902cffa544bd3&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset) [](https://app.codeant.ai/fix-in-ide?tool=vscode-claude&prompt_id=daf62a6903c04a0abf0902cffa544bd3&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset)
*(Use Cmd/Ctrl + Click for best experience)*
<details>
<summary><b>Prompt for AI Agent 🤖 </b></summary>
```mdx
This is a comment left during a code review.
**Path:** superset/common/grouping_sets.py
**Line:** 111:115
**Comment:**
*Custom Rule: Add a concrete type annotation to this local DataFrame variable assignment to keep local variable typing explicit.
Validate the correctness of the flagged issue. If correct, How can I resolve this? If you propose a fix, implement it and please make it concise.
Once fix is implemented, also check other comments on the same PR, and ask user if the user wants to fix the rest of the comments as well. if said yes, then fetch all the comments validate the correctness and implement a minimal fix
```
</details>
<a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F41184&comment_hash=15766b4510bf1c5abcdcec23d3f9b71aa9f7e0f79084c907cd6b90e673c5a58e&reaction=like'>👍</a> | <a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F41184&comment_hash=15766b4510bf1c5abcdcec23d3f9b71aa9f7e0f79084c907cd6b90e673c5a58e&reaction=dislike'>👎</a>
##########
superset-frontend/plugins/plugin-chart-pivot-table/test/plugin/utilities.test.ts:
##########
@@ -0,0 +1,446 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import { TimeGranularity } from '@superset-ui/core';
+import buildGroupbyCombinations, {
+ isAdditiveMetric,
+ allMetricsAdditive,
+ additiveReducerFor,
+ synthesizeAdditiveLevels,
+ splitGroupingSetsResult,
+ groupingMarkerLabel,
+} from '../../src/plugin/utilities';
+import { PivotTableQueryFormData, MetricsLayoutEnum } from '../../src/types';
+
+const baseFormData = {
+ groupbyRows: ['row1', 'row2'],
+ groupbyColumns: ['col1', 'col2'],
+ metrics: ['metric1', 'metric2'],
+ tableRenderer: 'Table With Subtotal',
+ colOrder: 'key_a_to_z',
+ rowOrder: 'key_a_to_z',
+ aggregateFunction: 'Sum',
+ metricsLayout: MetricsLayoutEnum.ROWS,
+ transposePivot: false,
+ rowSubtotalPosition: true,
+ colSubtotalPosition: true,
+ colTotals: true,
+ colSubTotals: true,
+ rowTotals: true,
+ rowSubTotals: true,
+ valueFormat: 'SMART_NUMBER',
+ datasource: '5__table',
+ viz_type: 'my_chart',
+ width: 800,
+ height: 600,
+ combineMetric: false,
+ verboseMap: {},
+ columnFormats: {},
+ currencyFormats: {},
+ metricColorFormatters: [],
+ dateFormatters: {},
+ setDataMask: () => {},
+ legacy_order_by: 'count',
+ order_desc: true,
+ margin: 0,
+ time_grain_sqla: TimeGranularity.MONTH,
+ temporal_columns_lookup: { col1: true },
+ currencyFormat: { symbol: 'USD', symbolPosition: 'prefix' },
+} as unknown as PivotTableQueryFormData;
+
+test('should build all combinations for basic pivot table', () => {
+ const combinations = buildGroupbyCombinations(baseFormData);
+
+ expect(combinations).toEqual([
+ { rows: [], columns: [] },
+ { rows: [], columns: ['col1'] },
+ { rows: [], columns: ['col1', 'col2'] },
+ { rows: ['row1'], columns: [] },
+ { rows: ['row1'], columns: ['col1'] },
+ { rows: ['row1'], columns: ['col1', 'col2'] },
+ { rows: ['row1', 'row2'], columns: [] },
+ { rows: ['row1', 'row2'], columns: ['col1'] },
+ { rows: ['row1', 'row2'], columns: ['col1', 'col2'] },
+ ]);
+
+ expect(combinations).toHaveLength(9);
+});
+
+test('should handle transposed pivot correctly', () => {
+ const modifiedFormData = {
+ ...baseFormData,
+ transposePivot: true,
+ };
+
+ const combinations = buildGroupbyCombinations(modifiedFormData);
+
+ expect(combinations).toEqual([
+ { rows: [], columns: [] },
+ { rows: [], columns: ['row1'] },
+ { rows: [], columns: ['row1', 'row2'] },
+ { rows: ['col1'], columns: [] },
+ { rows: ['col1'], columns: ['row1'] },
+ { rows: ['col1'], columns: ['row1', 'row2'] },
+ { rows: ['col1', 'col2'], columns: [] },
+ { rows: ['col1', 'col2'], columns: ['row1'] },
+ { rows: ['col1', 'col2'], columns: ['row1', 'row2'] },
+ ]);
+});
+
+test('should filter combinations when combineMetric is true with ROWS layout', () => {
+ const modifiedFormData = {
+ ...baseFormData,
+ combineMetric: true,
+ metricsLayout: MetricsLayoutEnum.ROWS,
+ };
+
+ const combinations = buildGroupbyCombinations(modifiedFormData);
+
+ expect(combinations).toEqual([
+ { rows: ['row1', 'row2'], columns: [] },
+ { rows: ['row1', 'row2'], columns: ['col1'] },
+ { rows: ['row1', 'row2'], columns: ['col1', 'col2'] },
+ ]);
+
+ expect(combinations).toHaveLength(3);
+});
+
+test('should filter combinations when combineMetric is true with COLUMNS layout', () => {
+ const modifiedFormData = {
+ ...baseFormData,
+ combineMetric: true,
+ metricsLayout: MetricsLayoutEnum.COLUMNS,
+ };
+
+ const combinations = buildGroupbyCombinations(modifiedFormData);
+
+ expect(combinations).toEqual([
+ { rows: [], columns: ['col1', 'col2'] },
+ { rows: ['row1'], columns: ['col1', 'col2'] },
+ { rows: ['row1', 'row2'], columns: ['col1', 'col2'] },
+ ]);
+
+ expect(combinations).toHaveLength(3);
+});
+
+test('should handle single dimension in rows only', () => {
+ const modifiedFormData = {
+ ...baseFormData,
+ groupbyRows: ['row'],
+ groupbyColumns: [],
+ };
+
+ const combinations = buildGroupbyCombinations(modifiedFormData);
+
+ expect(combinations).toEqual([
+ { rows: [], columns: [] },
+ { rows: ['row'], columns: [] },
+ ]);
+
+ expect(combinations).toHaveLength(2);
+});
+
+test('should handle single dimension in columns only', () => {
+ const modifiedFormData = {
+ ...baseFormData,
+ groupbyRows: [],
+ groupbyColumns: ['col'],
+ };
+
+ const combinations = buildGroupbyCombinations(modifiedFormData);
+
+ expect(combinations).toEqual([
+ { rows: [], columns: [] },
+ { rows: [], columns: ['col'] },
+ ]);
+
+ expect(combinations).toHaveLength(2);
+});
+
+test('should handle empty groupby arrays', () => {
+ const modifiedFormData = {
+ ...baseFormData,
+ groupbyRows: [],
+ groupbyColumns: [],
+ };
+
+ const combinations = buildGroupbyCombinations(modifiedFormData);
+
+ expect(combinations).toEqual([{ rows: [], columns: [] }]);
+
+ expect(combinations).toHaveLength(1);
+});
+
+test('should work with combineMetric and transposed pivot', () => {
+ const modifiedFormData = {
+ ...baseFormData,
+ transposePivot: true,
+ combineMetric: true,
+ metricsLayout: MetricsLayoutEnum.COLUMNS,
+ };
+
+ const combinations = buildGroupbyCombinations(modifiedFormData);
+
+ expect(combinations).toEqual([
+ { rows: [], columns: ['row1', 'row2'] },
+ { rows: ['col1'], columns: ['row1', 'row2'] },
+ { rows: ['col1', 'col2'], columns: ['row1', 'row2'] },
+ ]);
+});
+
+test('should handle combineMetric with empty arrays correctly', () => {
+ const modifiedFormData = {
+ ...baseFormData,
+ groupbyRows: [],
+ groupbyColumns: ['col'],
+ combineMetric: true,
+ metricsLayout: MetricsLayoutEnum.ROWS,
+ };
+
+ const combinations = buildGroupbyCombinations(modifiedFormData);
+
+ expect(combinations).toEqual([
+ { rows: [], columns: [] },
+ { rows: [], columns: ['col'] },
+ ]);
+});
+
+test('should work with large number of dimensions', () => {
+ const modifiedFormData = {
+ ...baseFormData,
+ groupbyRows: ['r1', 'r2', 'r3', 'r4'],
+ groupbyColumns: ['c1', 'c2', 'c3'],
+ };
+
+ const combinations = buildGroupbyCombinations(modifiedFormData);
+
+ expect(combinations).toHaveLength(20);
+
+ expect(combinations).toContainEqual({ rows: [], columns: [] });
+ expect(combinations).toContainEqual({
+ rows: ['r1', 'r2', 'r3', 'r4'],
+ columns: ['c1', 'c2', 'c3'],
+ });
+});
+
+test('isAdditiveMetric: SIMPLE metrics with additive aggregates are additive', () => {
+ expect(
+ isAdditiveMetric({
+ expressionType: 'SIMPLE',
+ aggregate: 'SUM',
+ column: { column_name: 'num' },
+ label: 'sum_num',
+ } as any),
+ ).toBe(true);
+ expect(
+ isAdditiveMetric({
+ expressionType: 'SIMPLE',
+ aggregate: 'COUNT',
+ column: { column_name: 'num' },
+ label: 'count_num',
+ } as any),
+ ).toBe(true);
+});
+
+test('isAdditiveMetric: non-additive aggregates, SQL, and saved metrics are not additive', () => {
+ expect(
+ isAdditiveMetric({
+ expressionType: 'SIMPLE',
+ aggregate: 'AVG',
+ column: { column_name: 'num' },
+ label: 'avg_num',
+ } as any),
+ ).toBe(false);
Review Comment:
**Suggestion:** Replace this `as any` cast with the appropriate typed metric shape for a non-additive aggregate case. [custom_rule]
**Severity Level:** Minor 🧹
<details>
<summary><b>Why it matters? ⭐ </b></summary>
The code in the final file contains an `as any` cast here, so the no-any-types rule is actually violated.
</details>
<details>
<summary><b>Rule source 📖 </b></summary>
.cursor/rules/dev-standard.mdc (line 16)
</details>
[](https://app.codeant.ai/fix-in-ide?tool=cursor&prompt_id=ff37acb4268e4e42a884813648358ba4&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset) [](https://app.codeant.ai/fix-in-ide?tool=vscode-claude&prompt_id=ff37acb4268e4e42a884813648358ba4&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset)
*(Use Cmd/Ctrl + Click for best experience)*
<details>
<summary><b>Prompt for AI Agent 🤖 </b></summary>
```mdx
This is a comment left during a code review.
**Path:** superset-frontend/plugins/plugin-chart-pivot-table/test/plugin/utilities.test.ts
**Line:** 262:269
**Comment:**
*Custom Rule: Replace this `as any` cast with the appropriate typed metric shape for a non-additive aggregate case.
Validate the correctness of the flagged issue. If correct, How can I resolve this? If you propose a fix, implement it and please make it concise.
Once fix is implemented, also check other comments on the same PR, and ask user if the user wants to fix the rest of the comments as well. if said yes, then fetch all the comments validate the correctness and implement a minimal fix
```
</details>
<a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F41184&comment_hash=fb81fed90482cb5b41e0cc07b221ef15e7676d153a58a2cffb0de66d2243367b&reaction=like'>👍</a> | <a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F41184&comment_hash=fb81fed90482cb5b41e0cc07b221ef15e7676d153a58a2cffb0de66d2243367b&reaction=dislike'>👎</a>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(table/pivot-table): correct non-additive totals/subtotals via DB rollup [POC for SIP-216] [superset]
Posted by "bito-code-review[bot] (via GitHub)" <gi...@apache.org>.
bito-code-review[bot] commented on PR #41184:
URL: https://github.com/apache/superset/pull/41184#issuecomment-4935144078
AI Code Review is in progress (usually takes 3 to 15 minutes unless it's a very large PR).
<details>
<summary>
Bito Usage Guide
</summary>
**Commands**
Type the following command in the pull request comment and save the comment.
- `/review` - Manually triggers a full AI review.
- `/pause` - Pauses automatic reviews on this pull request.
- `/resume` - Resumes automatic reviews.
- `/resolve` - Marks all Bito-posted review comments as resolved.
- `/abort` - Cancels all in-progress reviews.
Refer to the <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/available-commands" target="_blank" rel="noopener noreferrer">documentation</a> for additional commands.
**Configuration**
This repository uses `Superset` You can customize the agent settings <a href="https://alpha.bito.ai/home/ai-agents/code-review-agent" target="_blank" rel="noopener noreferrer">here</a> or contact your Bito workspace admin at evan@preset.io.
**Documentation & Help**
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/getting-started/install-run-using-bito-cloud/create-or-customize-an-agent-instance" target="_blank" rel="noopener noreferrer">Customize agent settings</a>
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/implementing-custom-code-review-rules" target="_blank" rel="noopener noreferrer">Review rules</a>
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent" target="_blank" rel="noopener noreferrer">General documentation</a>
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/faqs" target="_blank" rel="noopener noreferrer">FAQ</a>
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(table/pivot-table): correct non-additive totals/subtotals via DB rollup [POC for SIP-216] [superset]
Posted by "codeant-ai-for-open-source[bot] (via GitHub)" <gi...@apache.org>.
codeant-ai-for-open-source[bot] commented on code in PR #41184:
URL: https://github.com/apache/superset/pull/41184#discussion_r3562164380
##########
superset/db_engine_specs/presto.py:
##########
@@ -166,6 +166,7 @@ class PrestoBaseEngineSpec(BaseEngineSpec, metaclass=ABCMeta):
supports_dynamic_schema = True
supports_catalog = supports_dynamic_catalog = supports_cross_catalog_queries = True
+ supports_grouping_sets = True
Review Comment:
**Suggestion:** Add an explicit type annotation for this newly introduced class attribute so the boolean capability flag is type-safe and compliant with the typing rule. [custom_rule]
**Severity Level:** Minor 🧹
<details>
<summary><b>Why it matters? ⭐ </b></summary>
This newly added class attribute is a boolean configuration value that can be annotated (for example, `supports_grouping_sets: bool = True`). Since it is added without a type hint, it matches the Python type-hint rule.
</details>
<details>
<summary><b>Rule source 📖 </b></summary>
.cursor/rules/dev-standard.mdc (line 28)
</details>
[](https://app.codeant.ai/fix-in-ide?tool=cursor&prompt_id=2ec64881f13f4328964d8133c44e849b&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset) [](https://app.codeant.ai/fix-in-ide?tool=vscode-claude&prompt_id=2ec64881f13f4328964d8133c44e849b&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset)
*(Use Cmd/Ctrl + Click for best experience)*
<details>
<summary><b>Prompt for AI Agent 🤖 </b></summary>
```mdx
This is a comment left during a code review.
**Path:** superset/db_engine_specs/presto.py
**Line:** 169:169
**Comment:**
*Custom Rule: Add an explicit type annotation for this newly introduced class attribute so the boolean capability flag is type-safe and compliant with the typing rule.
Validate the correctness of the flagged issue. If correct, How can I resolve this? If you propose a fix, implement it and please make it concise.
Once fix is implemented, also check other comments on the same PR, and ask user if the user wants to fix the rest of the comments as well. if said yes, then fetch all the comments validate the correctness and implement a minimal fix
```
</details>
<a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F41184&comment_hash=b31ff590d8c4a17f3088e7e120798706475261d85a30e44de7e364b800d976ba&reaction=like'>👍</a> | <a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F41184&comment_hash=b31ff590d8c4a17f3088e7e120798706475261d85a30e44de7e364b800d976ba&reaction=dislike'>👎</a>
##########
superset/common/query_context_processor.py:
##########
@@ -252,9 +255,58 @@ def get_query_result(self, query_object: QueryObject) -> QueryResult:
This method delegates to the datasource's get_query_result method,
which handles query execution, normalization, time offsets, and
post-processing.
+
+ When the query requests rollup ``grouping_sets`` but the engine does not
+ support native ``GROUPING SETS``, fall back to one query per level and
+ concatenate the results with ``GROUPING()``-equivalent markers, so the
+ combined result matches the shape the native path produces (SIP.md,
+ phase 3b). Engines that support it run the single native query.
"""
+ if query_object.grouping_sets and not self._supports_grouping_sets():
+ return self._grouping_sets_fallback(query_object)
return self._qc_datasource.get_query_result(query_object)
+ def _supports_grouping_sets(self) -> bool:
+ engine_spec = getattr(self._qc_datasource, "db_engine_spec", None)
Review Comment:
**Suggestion:** Add an explicit type annotation for this local variable so the engine spec contract is clear and statically checkable. [custom_rule]
**Severity Level:** Minor 🧹
<details>
<summary><b>Why it matters? ⭐ </b></summary>
This is a new local variable in modified Python code and it is type-annotatable. The code omits an explicit type hint, which matches the rule requiring type hints for relevant variables that can be annotated.
</details>
<details>
<summary><b>Rule source 📖 </b></summary>
.cursor/rules/dev-standard.mdc (line 28)
</details>
[](https://app.codeant.ai/fix-in-ide?tool=cursor&prompt_id=d31ec908143245968aa170d490ac9ed4&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset) [](https://app.codeant.ai/fix-in-ide?tool=vscode-claude&prompt_id=d31ec908143245968aa170d490ac9ed4&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset)
*(Use Cmd/Ctrl + Click for best experience)*
<details>
<summary><b>Prompt for AI Agent 🤖 </b></summary>
```mdx
This is a comment left during a code review.
**Path:** superset/common/query_context_processor.py
**Line:** 270:270
**Comment:**
*Custom Rule: Add an explicit type annotation for this local variable so the engine spec contract is clear and statically checkable.
Validate the correctness of the flagged issue. If correct, How can I resolve this? If you propose a fix, implement it and please make it concise.
Once fix is implemented, also check other comments on the same PR, and ask user if the user wants to fix the rest of the comments as well. if said yes, then fetch all the comments validate the correctness and implement a minimal fix
```
</details>
<a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F41184&comment_hash=2c3afc7744f09839ebc3eb11a02c1d073d948449e6fec8dd2cbb831db78e10a8&reaction=like'>👍</a> | <a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F41184&comment_hash=2c3afc7744f09839ebc3eb11a02c1d073d948449e6fec8dd2cbb831db78e10a8&reaction=dislike'>👎</a>
##########
superset/common/query_context_processor.py:
##########
@@ -252,9 +255,58 @@ def get_query_result(self, query_object: QueryObject) -> QueryResult:
This method delegates to the datasource's get_query_result method,
which handles query execution, normalization, time offsets, and
post-processing.
+
+ When the query requests rollup ``grouping_sets`` but the engine does not
+ support native ``GROUPING SETS``, fall back to one query per level and
+ concatenate the results with ``GROUPING()``-equivalent markers, so the
+ combined result matches the shape the native path produces (SIP.md,
+ phase 3b). Engines that support it run the single native query.
"""
+ if query_object.grouping_sets and not self._supports_grouping_sets():
+ return self._grouping_sets_fallback(query_object)
return self._qc_datasource.get_query_result(query_object)
+ def _supports_grouping_sets(self) -> bool:
+ engine_spec = getattr(self._qc_datasource, "db_engine_spec", None)
+ return bool(engine_spec and engine_spec.supports_grouping_sets)
+
+ def _grouping_sets_fallback(self, query_object: QueryObject) -> QueryResult:
+ """
+ Emulate a GROUPING SETS query on engines without native support: run one
+ query per rollup level and concatenate, tagging each level's rows with
+ the same per-column markers the native path emits.
+ """
+ levels = query_object.grouping_sets
Review Comment:
**Suggestion:** Add a concrete type annotation for this variable to document the expected grouping set structure and improve type safety in downstream loop logic. [custom_rule]
**Severity Level:** Minor 🧹
<details>
<summary><b>Why it matters? ⭐ </b></summary>
This modified Python code introduces an unannotated local variable whose structure is used in the loop below. Since the variable is clearly type-annotatable, the omission aligns with the type-hints rule.
</details>
<details>
<summary><b>Rule source 📖 </b></summary>
.cursor/rules/dev-standard.mdc (line 28)
</details>
[](https://app.codeant.ai/fix-in-ide?tool=cursor&prompt_id=58631b59279648debb4b482873c04677&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset) [](https://app.codeant.ai/fix-in-ide?tool=vscode-claude&prompt_id=58631b59279648debb4b482873c04677&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset)
*(Use Cmd/Ctrl + Click for best experience)*
<details>
<summary><b>Prompt for AI Agent 🤖 </b></summary>
```mdx
This is a comment left during a code review.
**Path:** superset/common/query_context_processor.py
**Line:** 279:279
**Comment:**
*Custom Rule: Add a concrete type annotation for this variable to document the expected grouping set structure and improve type safety in downstream loop logic.
Validate the correctness of the flagged issue. If correct, How can I resolve this? If you propose a fix, implement it and please make it concise.
Once fix is implemented, also check other comments on the same PR, and ask user if the user wants to fix the rest of the comments as well. if said yes, then fetch all the comments validate the correctness and implement a minimal fix
```
</details>
<a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F41184&comment_hash=f5f61e902db660b961ed5ef5b884a539e3ea13c6b390a008d807f7b006be6428&reaction=like'>👍</a> | <a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F41184&comment_hash=f5f61e902db660b961ed5ef5b884a539e3ea13c6b390a008d807f7b006be6428&reaction=dislike'>👎</a>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(table/pivot-table): correct non-additive totals/subtotals via DB rollup [POC for SIP-216] [superset]
Posted by "rusackas (via GitHub)" <gi...@apache.org>.
rusackas commented on code in PR #41184:
URL: https://github.com/apache/superset/pull/41184#discussion_r3565612015
##########
superset/common/query_context_processor.py:
##########
@@ -252,9 +256,60 @@ def get_query_result(self, query_object: QueryObject) -> QueryResult:
This method delegates to the datasource's get_query_result method,
which handles query execution, normalization, time offsets, and
post-processing.
+
+ When the query requests rollup ``grouping_sets`` but the engine does not
+ support native ``GROUPING SETS``, fall back to one query per level and
+ concatenate the results with ``GROUPING()``-equivalent markers, so the
+ combined result matches the shape the native path produces (SIP.md,
+ phase 3b). Engines that support it run the single native query.
"""
+ if query_object.grouping_sets and not self._supports_grouping_sets():
+ return self._grouping_sets_fallback(query_object)
return self._qc_datasource.get_query_result(query_object)
+ def _supports_grouping_sets(self) -> bool:
+ engine_spec: BaseEngineSpec | None = getattr(
+ self._qc_datasource, "db_engine_spec", None
+ )
+ return bool(engine_spec and engine_spec.supports_grouping_sets)
+
+ def _grouping_sets_fallback(self, query_object: QueryObject) -> QueryResult:
+ """
+ Emulate a GROUPING SETS query on engines without native support: run one
+ query per rollup level and concatenate, tagging each level's rows with
+ the same per-column markers the native path emits.
+ """
+ levels: list[list[str]] = query_object.grouping_sets
+ # Use the same label derivation as the native path (physical column name
+ # or adhoc column label) so both column kinds are represented and each
+ # label maps back to its own column, in the same order as the source
+ # list.
+ all_labels: list[str] = [get_column_name(col) for col in query_object.columns]
+ label_to_column = dict(zip(all_labels, query_object.columns, strict=True))
Review Comment:
Added the type hint.
##########
superset/common/query_context_processor.py:
##########
@@ -252,9 +256,60 @@ def get_query_result(self, query_object: QueryObject) -> QueryResult:
This method delegates to the datasource's get_query_result method,
which handles query execution, normalization, time offsets, and
post-processing.
+
+ When the query requests rollup ``grouping_sets`` but the engine does not
+ support native ``GROUPING SETS``, fall back to one query per level and
+ concatenate the results with ``GROUPING()``-equivalent markers, so the
+ combined result matches the shape the native path produces (SIP.md,
+ phase 3b). Engines that support it run the single native query.
"""
+ if query_object.grouping_sets and not self._supports_grouping_sets():
+ return self._grouping_sets_fallback(query_object)
return self._qc_datasource.get_query_result(query_object)
+ def _supports_grouping_sets(self) -> bool:
+ engine_spec: BaseEngineSpec | None = getattr(
+ self._qc_datasource, "db_engine_spec", None
+ )
+ return bool(engine_spec and engine_spec.supports_grouping_sets)
+
+ def _grouping_sets_fallback(self, query_object: QueryObject) -> QueryResult:
+ """
+ Emulate a GROUPING SETS query on engines without native support: run one
+ query per rollup level and concatenate, tagging each level's rows with
+ the same per-column markers the native path emits.
+ """
+ levels: list[list[str]] = query_object.grouping_sets
+ # Use the same label derivation as the native path (physical column name
+ # or adhoc column label) so both column kinds are represented and each
+ # label maps back to its own column, in the same order as the source
+ # list.
+ all_labels: list[str] = [get_column_name(col) for col in query_object.columns]
+ label_to_column = dict(zip(all_labels, query_object.columns, strict=True))
+
+ frames: list[pd.DataFrame] = []
+ result: QueryResult | None = None
+ for level in levels:
+ level_labels = set(level)
Review Comment:
Added the type hint.
##########
superset-frontend/plugins/plugin-chart-pivot-table/test/plugin/transformProps.test.ts:
##########
@@ -64,36 +63,73 @@ const chartProps = new ChartProps<QueryFormData>({
theme: supersetTheme,
Review Comment:
Fixed, typed as `QueryData` instead.
##########
superset-frontend/plugins/plugin-chart-pivot-table/test/plugin/transformProps.test.ts:
##########
@@ -64,36 +63,73 @@ const chartProps = new ChartProps<QueryFormData>({
theme: supersetTheme,
});
-test('should transform chart props for viz', () => {
- expect(transformProps(chartProps)).toEqual({
- width: 800,
- height: 600,
- groupbyRows: ['row1', 'row2'],
- groupbyColumns: ['col1', 'col2'],
- metrics: ['metric1', 'metric2'],
- tableRenderer: 'Table With Subtotal',
- colOrder: 'key_a_to_z',
- rowOrder: 'key_a_to_z',
- aggregateFunction: 'Sum',
- transposePivot: true,
- combineMetric: true,
- rowSubtotalPosition: true,
- colSubtotalPosition: true,
+test('should pass through formData props for viz', () => {
+ const result = transformProps(chartProps) as any;
+ expect(result.width).toBe(800);
+ expect(result.height).toBe(600);
+ expect(result.groupbyRows).toEqual(['row1', 'row2']);
+ expect(result.groupbyColumns).toEqual(['col1', 'col2']);
+ expect(result.metrics).toEqual(['metric1', 'metric2']);
+ expect(result.metricsLayout).toBe(MetricsLayoutEnum.COLUMNS);
+ expect(result.currencyFormat).toEqual({
Review Comment:
Fixed, typed as `QueryData` instead.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(table/pivot-table): correct non-additive totals/subtotals via DB rollup [POC for SIP-216] [superset]
Posted by "codeant-ai-for-open-source[bot] (via GitHub)" <gi...@apache.org>.
codeant-ai-for-open-source[bot] commented on code in PR #41184:
URL: https://github.com/apache/superset/pull/41184#discussion_r3564138579
##########
superset/common/query_context_processor.py:
##########
@@ -252,9 +256,60 @@ def get_query_result(self, query_object: QueryObject) -> QueryResult:
This method delegates to the datasource's get_query_result method,
which handles query execution, normalization, time offsets, and
post-processing.
+
+ When the query requests rollup ``grouping_sets`` but the engine does not
+ support native ``GROUPING SETS``, fall back to one query per level and
+ concatenate the results with ``GROUPING()``-equivalent markers, so the
+ combined result matches the shape the native path produces (SIP.md,
+ phase 3b). Engines that support it run the single native query.
"""
+ if query_object.grouping_sets and not self._supports_grouping_sets():
+ return self._grouping_sets_fallback(query_object)
return self._qc_datasource.get_query_result(query_object)
+ def _supports_grouping_sets(self) -> bool:
+ engine_spec: BaseEngineSpec | None = getattr(
+ self._qc_datasource, "db_engine_spec", None
+ )
+ return bool(engine_spec and engine_spec.supports_grouping_sets)
+
+ def _grouping_sets_fallback(self, query_object: QueryObject) -> QueryResult:
+ """
+ Emulate a GROUPING SETS query on engines without native support: run one
+ query per rollup level and concatenate, tagging each level's rows with
+ the same per-column markers the native path emits.
+ """
+ levels: list[list[str]] = query_object.grouping_sets
+ # Use the same label derivation as the native path (physical column name
+ # or adhoc column label) so both column kinds are represented and each
+ # label maps back to its own column, in the same order as the source
+ # list.
+ all_labels = [get_column_name(col) for col in query_object.columns]
+ label_to_column = dict(zip(all_labels, query_object.columns, strict=True))
+
+ frames: list[pd.DataFrame] = []
+ result: QueryResult | None = None
+ for level in levels:
+ level_labels = set(level)
+ sub_query = copy.copy(query_object)
+ sub_query.grouping_sets = []
+ sub_query.columns = [
+ label_to_column[label] for label in all_labels if label in level_labels
+ ]
Review Comment:
**Suggestion:** The fallback reuses the original query object (including `row_limit`) for every rollup level. That applies the limit independently per level, which can truncate subtotal/grand-total rows and produce different results than the native GROUPING SETS path (which explicitly skips per-level limiting). Clear `row_limit` on each fallback subquery and, if needed, enforce any final cap only once on the concatenated result. [logic error]
<details>
<summary><b>Severity Level:</b> Critical 🚨</summary>
```mdx
❌ Pivot/pivot-table rollup totals incorrect on non-supporting engines.
⚠️ Grouping-sets rollup results inconsistent across database backends.
```
</details>
<details>
<summary><b>Steps of Reproduction ✅ </b></summary>
```mdx
1. Configure a pivot/table Chart Data payload with a grouping_sets rollup on a dataset
backed by an engine without native GROUPING SETS support
(SqliteEngineSpec.supports_grouping_sets is False, verified in
tests/unit_tests/db_engine_specs/test_grouping_sets_capability.py:39). Use a payload
similar to TestGroupingSetsRollup in
tests/integration_tests/non_additive_totals_tests.py:230-249, but targeting such an engine
and with a finite row_limit (for example 100).
2. The ChartData API parses this payload via ChartDataQueryContextSchema().load(payload)
as in tests/integration_tests/non_additive_totals_tests.py:42-45, producing a QueryContext
whose first QueryObject has columns ["gender", "state"], a non-additive metric,
grouping_sets levels, and row_limit set to the requested value.
3. When QueryContext.get_query_result(query_context.queries[0]) is called
(tests/integration_tests/non_additive_totals_tests.py:299-2), it delegates to
QueryContextProcessor.get_query_result(query_object) in
superset/common/query_context_processor.py:33-49. Because
datasource.db_engine_spec.supports_grouping_sets is False, _supports_grouping_sets()
(lines 51-55) returns False and _grouping_sets_fallback(query_object) (lines 57-92) is
invoked.
4. Inside _grouping_sets_fallback, the loop over levels at
superset/common/query_context_processor.py:73-80 creates sub_query =
copy.copy(query_object), leaves sub_query.row_limit unchanged, and calls
self._qc_datasource.get_query_result(sub_query) for each level. This applies the same
row_limit independently to every level, truncating each per-level DataFrame before
concatenation at line 91. The combined result.df therefore omits rows that would be
present if the GROUPING SETS path fetched all rows once and applied any limit only on the
final combined result, yielding incorrect subtotals/grand totals on engines using the
fallback.
```
</details>
[](https://app.codeant.ai/fix-in-ide?tool=cursor&prompt_id=3a51977983a74a1c9ef6c6c365e0c1df&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset) [](https://app.codeant.ai/fix-in-ide?tool=vscode-claude&prompt_id=3a51977983a74a1c9ef6c6c365e0c1df&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset)
*(Use Cmd/Ctrl + Click for best experience)*
<details>
<summary><b>Prompt for AI Agent 🤖 </b></summary>
```mdx
This is a comment left during a code review.
**Path:** superset/common/query_context_processor.py
**Line:** 294:298
**Comment:**
*Logic Error: The fallback reuses the original query object (including `row_limit`) for every rollup level. That applies the limit independently per level, which can truncate subtotal/grand-total rows and produce different results than the native GROUPING SETS path (which explicitly skips per-level limiting). Clear `row_limit` on each fallback subquery and, if needed, enforce any final cap only once on the concatenated result.
Validate the correctness of the flagged issue. If correct, How can I resolve this? If you propose a fix, implement it and please make it concise.
Once fix is implemented, also check other comments on the same PR, and ask user if the user wants to fix the rest of the comments as well. if said yes, then fetch all the comments validate the correctness and implement a minimal fix
```
</details>
<a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F41184&comment_hash=6478106458fb985c103eac66d2af066b06ec84f9e2d6fe063e2891882902af20&reaction=like'>👍</a> | <a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F41184&comment_hash=6478106458fb985c103eac66d2af066b06ec84f9e2d6fe063e2891882902af20&reaction=dislike'>👎</a>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(table/pivot-table): correct non-additive totals/subtotals via DB rollup [POC for SIP-216] [superset]
Posted by "codeant-ai-for-open-source[bot] (via GitHub)" <gi...@apache.org>.
codeant-ai-for-open-source[bot] commented on code in PR #41184:
URL: https://github.com/apache/superset/pull/41184#discussion_r3565632155
##########
superset/common/query_context_processor.py:
##########
@@ -252,9 +256,69 @@ def get_query_result(self, query_object: QueryObject) -> QueryResult:
This method delegates to the datasource's get_query_result method,
which handles query execution, normalization, time offsets, and
post-processing.
+
+ When the query requests rollup ``grouping_sets`` but the engine does not
+ support native ``GROUPING SETS``, fall back to one query per level and
+ concatenate the results with ``GROUPING()``-equivalent markers, so the
+ combined result matches the shape the native path produces (SIP.md,
+ phase 3b). Engines that support it run the single native query.
"""
+ if query_object.grouping_sets and not self._supports_grouping_sets():
+ return self._grouping_sets_fallback(query_object)
return self._qc_datasource.get_query_result(query_object)
+ def _supports_grouping_sets(self) -> bool:
+ engine_spec: BaseEngineSpec | None = getattr(
+ self._qc_datasource, "db_engine_spec", None
+ )
+ return bool(engine_spec and engine_spec.supports_grouping_sets)
+
+ def _grouping_sets_fallback(self, query_object: QueryObject) -> QueryResult:
+ """
+ Emulate a GROUPING SETS query on engines without native support: run one
+ query per rollup level and concatenate, tagging each level's rows with
+ the same per-column markers the native path emits.
+ """
+ levels: list[list[str]] = query_object.grouping_sets
+ # Use the same label derivation as the native path (physical column name
+ # or adhoc column label) so both column kinds are represented and each
+ # label maps back to its own column, in the same order as the source
+ # list.
+ all_labels: list[str] = [get_column_name(col) for col in query_object.columns]
+ label_to_column: dict[str, Column] = dict(
+ zip(all_labels, query_object.columns, strict=True)
+ )
+
+ frames: list[pd.DataFrame] = []
+ result: QueryResult | None = None
+ for level in levels:
+ level_labels: set[str] = set(level)
+ sub_query = copy.copy(query_object)
+ sub_query.grouping_sets = []
+ sub_query.columns = [
+ label_to_column[label] for label in all_labels if label in level_labels
+ ]
+ # A GROUPING SETS query computes a bounded set of rollup levels, so
+ # the native path never applies row_limit to it (see the
+ # `use_grouping_sets` check in models/helpers.py). Match that here:
+ # limiting each level's fallback sub-query independently would
+ # truncate subtotal/grand-total rows and diverge from the native
+ # result shape.
+ sub_query.row_limit = None
Review Comment:
**Suggestion:** The fallback path removes `row_limit` per level but leaves `row_offset` untouched, so each per-level subquery applies the offset independently. That diverges from native `GROUPING SETS` behavior (single global offset) and can silently drop subtotal/grand-total rows on paginated requests. Reset per-level `row_offset` and apply offset once after concatenation (or disable offset for this fallback path) to keep results consistent with native execution. [logic error]
<details>
<summary><b>Severity Level:</b> Major ⚠️</summary>
```mdx
- ❌ Grouping-sets rollup loses subtotal/grand total rows on pagination.
- ⚠️ Pivot and Table charts misrepresent non-additive totals when paginated.
- ⚠️ Engines without GROUPING SETS behave inconsistently versus supported engines.
```
</details>
<details>
<summary><b>Steps of Reproduction ✅ </b></summary>
```mdx
1. In `superset/tests/integration_tests/non_additive_totals_tests.py` inside
`TestGroupingSetsRollup.test_grouping_sets_returns_all_levels` (around lines 70-89),
extend the existing query payload by adding a non-zero offset, for example:
- set `payload["queries"][0]["row_offset"] = 1` alongside the existing `row_limit` and
`grouping_sets` fields.
2. Observe that this test constructs a `QueryContext` via
`ChartDataQueryContextSchema().load(payload)` at lines 100-101 of the same file:
- `query_context: QueryContext = ChartDataQueryContextSchema().load(payload)`
- `df = query_context.get_query_result(query_context.queries[0]).df`
This calls `QueryContext.get_query_result` in
`superset/common/query_context.py:134-135`, which delegates directly to
`QueryContextProcessor.get_query_result`.
3. In `superset/common/query_context_processor.py`,
`QueryContextProcessor.get_query_result` (lines 252-268) checks for rollup queries:
- If `query_object.grouping_sets` is non-empty and `_supports_grouping_sets()` is
False, it calls `self._grouping_sets_fallback(query_object)` at line 267.
- `_supports_grouping_sets` (lines 270-274) looks up
`self._qc_datasource.db_engine_spec.supports_grouping_sets`.
For the birth_names test dataset, the engine spec inherits from `BaseEngineSpec` where
`supports_grouping_sets = False` by default (`superset/db_engine_specs/base.py:567`),
so the fallback path is taken for engines like SQLite that lack native GROUPING SETS.
4. Inside `_grouping_sets_fallback` in
`superset/common/query_context_processor.py:282-61`, for each rollup `level`:
- A shallow copy of the original `QueryObject` is created: `sub_query =
copy.copy(query_object)` at line 37.
- `sub_query.grouping_sets` is cleared and `sub_query.columns` is restricted to that
level (lines 38-41).
- Crucially, `sub_query.row_offset` is not changed; it keeps the original non-zero
value set in the payload via `QueryObject.__init__`
(`superset/common/query_object.py:37-39`, where `self.row_offset = row_offset or 0`).
- Only `sub_query.row_limit` is reset to `None` at line 48 (the PR line under review),
and then `self._qc_datasource.get_query_result(sub_query)` is called at line 49.
5. The datasource’s SQL construction in `superset/models/helpers.py:3990-4016` shows how
pagination is applied:
- `use_grouping_sets` is computed earlier (lines 7-16 of the first BulkRead snippet)
and is False for these per-level fallback queries because `sub_query.grouping_sets` was
cleared.
- Row limit is only applied when `row_limit and not use_grouping_sets` (lines 10-14),
but row offset is always applied when non-zero: `if row_offset: qry =
qry.offset(row_offset)` at line 15.
Therefore, every per-level fallback query runs with the same non-zero `row_offset`
inherited from the original query object.
6. For rollup levels with very few rows (especially the grand-total level `[]` which
produces exactly one row in `TestGroupingSetsRollup`), applying `OFFSET 1` at the
sub-query level drops that entire level:
- The grand-total sub-query returns an empty `DataFrame` because all rows have been
skipped before aggregation.
- `_grouping_sets_fallback` still tags each per-level frame with
`grouping_marker_label` columns (lines 50-54) and concatenates them into `result.df`
via `pd.concat(frames, ignore_index=True)` at line 60, but the grand-total and possibly
some subtotal frames contribute no rows.
7. Downstream, the test splits the combined result back into per-level frames using
`split_grouping_sets_result` from `superset/common/grouping_sets.py:82-117`:
- Lines 107-116 of `non_additive_totals_tests.py` call `leaf, gender_sub, grand =
split_grouping_sets_result(df, levels, ["gender", "state"])`.
- The test then asserts `assert len(grand) == 1` and checks that the grand total ratio
is valid (lines 110-114).
Under the fallback with per-level offsets, `grand` becomes empty (`len(grand) == 0`)
and these assertions fail, demonstrating that subtotal/grand-total rows have been
silently dropped by the per-level row_offset.
8. Contrast this with the native GROUPING SETS path: for an engine where
`supports_grouping_sets = True`, such as Presto (`superset/db_engine_specs/presto.py:919`)
or Postgres (`superset/db_engine_specs/postgres.py:281`), the same payload builds a single
GROUPING SETS query in `models/helpers.py`:
- `use_grouping_sets` is True, so row limit is skipped for that query (`if row_limit
and not use_grouping_sets`), but a single global `row_offset` is still applied once at
the end (`if row_offset: qry = qry.offset(row_offset)`).
- The grand-total row remains present after offsetting, so `split_grouping_sets_result`
still finds `len(grand) == 1`.
This shows that the fallback path’s per-level offset semantics are inconsistent with
the native GROUPING SETS behavior and can drop rollup rows only when the fallback is
used.
```
</details>
[](https://app.codeant.ai/fix-in-ide?tool=cursor&prompt_id=0f4d1905d6154bd7a71f79c2c879acc4&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset) [](https://app.codeant.ai/fix-in-ide?tool=vscode-claude&prompt_id=0f4d1905d6154bd7a71f79c2c879acc4&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset)
*(Use Cmd/Ctrl + Click for best experience)*
<details>
<summary><b>Prompt for AI Agent 🤖 </b></summary>
```mdx
This is a comment left during a code review.
**Path:** superset/common/query_context_processor.py
**Line:** 307:307
**Comment:**
*Logic Error: The fallback path removes `row_limit` per level but leaves `row_offset` untouched, so each per-level subquery applies the offset independently. That diverges from native `GROUPING SETS` behavior (single global offset) and can silently drop subtotal/grand-total rows on paginated requests. Reset per-level `row_offset` and apply offset once after concatenation (or disable offset for this fallback path) to keep results consistent with native execution.
Validate the correctness of the flagged issue. If correct, How can I resolve this? If you propose a fix, implement it and please make it concise.
Once fix is implemented, also check other comments on the same PR, and ask user if the user wants to fix the rest of the comments as well. if said yes, then fetch all the comments validate the correctness and implement a minimal fix
```
</details>
<a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F41184&comment_hash=e2fe2b2008ae46989420c80b9289fe06bb3a2233fb6d082979b8d3e318c50e8b&reaction=like'>👍</a> | <a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F41184&comment_hash=e2fe2b2008ae46989420c80b9289fe06bb3a2233fb6d082979b8d3e318c50e8b&reaction=dislike'>👎</a>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(table/pivot-table): correct non-additive totals/subtotals via DB rollup [POC for SIP-216] [superset]
Posted by "bito-code-review[bot] (via GitHub)" <gi...@apache.org>.
bito-code-review[bot] commented on code in PR #41184:
URL: https://github.com/apache/superset/pull/41184#discussion_r3565630511
##########
superset-frontend/plugins/plugin-chart-pivot-table/test/react-pivottable/tableRenders.test.tsx:
##########
@@ -149,18 +202,83 @@ test('TableRenderer renders col totals row when colTotals is enabled', () => {
test('TableRenderer renders grand total when both totals are enabled', () => {
const props = buildDefaultProps({
+ data: TAGGED_COUNT_DATA,
+ vals: ['value'],
tableOptions: { rowTotals: true, colTotals: true },
});
renderWithTheme(<TableRenderer {...props} />);
- // The grand total cell should show "4" (total record count).
+ // The grand total cell shows the DB-computed "4" (total record count).
const grandTotalCells = screen
.getAllByRole('gridcell')
.filter(cell => cell.classList.contains('pvtGrandTotal'));
expect(grandTotalCells.length).toBe(1);
expect(grandTotalCells[0]).toHaveTextContent('4');
Review Comment:
<!-- Bito Reply -->
The reviewer is correct that `toHaveTextContent` is not a substring match by default in the context of how `jest-dom` handles text content. While it does check if the provided string is contained within the element's text, it is sensitive to the exact string representation. If the formatter renders the value as '4.00' instead of '4', the assertion `expect(grandTotalCells[0]).toHaveTextContent('4')` will fail because the text content of the element will be '4.00', and '4' is not an exact match for '4.00'.
**superset-frontend/plugins/plugin-chart-pivot-table/test/react-pivottable/tableRenders.test.tsx**
```
// The grand total cell shows the DB-computed "4" (total record count).
const grandTotalCells = screen
.getAllByRole('gridcell')
.filter(cell => cell.classList.contains('pvtGrandTotal'));
expect(grandTotalCells.length).toBe(1);
expect(grandTotalCells[0]).toHaveTextContent('4');
```
##########
superset-frontend/plugins/plugin-chart-pivot-table/src/react-pivottable/utilities.ts:
##########
@@ -1088,76 +1117,108 @@ class PivotData {
}
processRecord(record: PivotRecord): void {
- // this code is called in a tight loop
+ // this code is called in a tight loop.
+ // Each record is tagged (in PivotTableChart) with `__rows`/`__columns`:
+ // the dimension labels of the rollup level that produced it. The database
+ // has already aggregated that level, so we place the value into exactly
+ // one slot and store it verbatim (no re-aggregation). A key shorter than
+ // the full dimension list identifies a subtotal level. Records without
+ // tags (e.g. direct unit-test construction) fall back to the full
+ // dimension lists, i.e. they are treated as leaf rows. The `__` prefix
+ // keeps these rollup tags from colliding with a real dataset column
+ // named `rows`/`columns`.
+ const recordRows = (record.__rows as unknown as string[]) ?? null;
+ const recordColumns = (record.__columns as unknown as string[]) ?? null;
+ const levelRows = recordRows ?? (this.props.rows as string[]);
+ const levelColumns = recordColumns ?? (this.props.cols as string[]);
+
const colKey: string[] = [];
const rowKey: string[] = [];
- (this.props.cols as string[]).forEach((col: string) => {
+ levelColumns.forEach((col: string) => {
colKey.push(col in record ? String(record[col]) : 'null');
});
- (this.props.rows as string[]).forEach((row: string) => {
+ levelRows.forEach((row: string) => {
rowKey.push(row in record ? String(record[row]) : 'null');
});
- this.allTotal.push(record);
-
- const rowStart = this.subtotals.rowEnabled ? 1 : Math.max(1, rowKey.length);
- const colStart = this.subtotals.colEnabled ? 1 : Math.max(1, colKey.length);
-
- let isRowSubtotal;
- let isColSubtotal;
- for (let ri = rowStart; ri <= rowKey.length; ri += 1) {
- isRowSubtotal = ri < rowKey.length;
- const fRowKey = rowKey.slice(0, ri);
- const flatRowKey = flatKey(fRowKey);
- if (!this.rowTotals[flatRowKey]) {
- this.rowKeys.push(fRowKey);
- this.rowTotals[flatRowKey] = this.getFormattedAggregator(
- record,
- rowKey,
- )(this, fRowKey, []);
+ const flatRowKey = flatKey(rowKey);
+ const flatColKey = flatKey(colKey);
+
+ const isColSubtotal = colKey.length < (this.props.cols as string[]).length;
+ const isRowSubtotal = rowKey.length < (this.props.rows as string[]).length;
+
+ // Register/create the column-total slot the first time this colKey is seen.
+ if (colKey.length > 0 && !this.colTotals[flatColKey]) {
+ if (!isColSubtotal || this.subtotals.colEnabled) {
+ this.colKeys.push(colKey);
}
- this.rowTotals[flatRowKey].push(record);
- this.rowTotals[flatRowKey].isSubtotal = isRowSubtotal;
+ this.colTotals[flatColKey] = this.getFormattedAggregator(record, colKey)(
+ this,
+ [],
+ colKey,
+ );
}
-
- for (let ci = colStart; ci <= colKey.length; ci += 1) {
- isColSubtotal = ci < colKey.length;
- const fColKey = colKey.slice(0, ci);
- const flatColKey = flatKey(fColKey);
- if (!this.colTotals[flatColKey]) {
- this.colKeys.push(fColKey);
- this.colTotals[flatColKey] = this.getFormattedAggregator(
- record,
+ // Register/create the row-total slot the first time this rowKey is seen.
+ if (rowKey.length > 0 && !this.rowTotals[flatRowKey]) {
+ if (!isRowSubtotal || this.subtotals.rowEnabled) {
+ this.rowKeys.push(rowKey);
+ }
+ this.rowTotals[flatRowKey] = this.getFormattedAggregator(record, rowKey)(
+ this,
+ rowKey,
+ [],
+ );
+ }
+ // Create the body-cell slot.
+ if (rowKey.length > 0 && colKey.length > 0) {
+ if (!this.tree[flatRowKey]) {
+ this.tree[flatRowKey] = {};
+ }
+ if (!this.tree[flatRowKey][flatColKey]) {
+ this.tree[flatRowKey][flatColKey] = this.getFormattedAggregator(record)(
+ this,
+ rowKey,
colKey,
- )(this, [], fColKey);
+ );
}
+ }
+
+ // Place the value in exactly one slot, determined by the level.
+ if (rowKey.length === 0 && colKey.length === 0) {
+ this.allTotal.push(record);
+ } else if (rowKey.length === 0) {
this.colTotals[flatColKey].push(record);
this.colTotals[flatColKey].isSubtotal = isColSubtotal;
+ } else if (colKey.length === 0) {
+ this.rowTotals[flatRowKey].push(record);
+ this.rowTotals[flatRowKey].isSubtotal = isRowSubtotal;
+ } else {
+ this.tree[flatRowKey][flatColKey].push(record);
+ this.tree[flatRowKey][flatColKey].isRowSubtotal = isRowSubtotal;
+ this.tree[flatRowKey][flatColKey].isColSubtotal = isColSubtotal;
+ this.tree[flatRowKey][flatColKey].isSubtotal =
+ isRowSubtotal || isColSubtotal;
}
- // And now fill in for all the sub-cells.
- for (let ri = rowStart; ri <= rowKey.length; ri += 1) {
- isRowSubtotal = ri < rowKey.length;
- const fRowKey = rowKey.slice(0, ri);
- const flatRowKey = flatKey(fRowKey);
- if (!this.tree[flatRowKey]) {
- this.tree[flatRowKey] = {};
+ // Metric-collapse totals. The metric is a pseudo-dimension always present on
+ // one axis, so no rollup level produces an empty key on that axis -- which
+ // would leave the opposite "Total" axis and the grand-total corner empty.
+ // When a record's axis holds only the metric (no real dims there), its value
+ // is also the collapsed total for that axis, so mirror it into rowTotals /
+ // colTotals / allTotal. (For a single metric this equals the metric column;
+ // for multiple metrics it is the last metric -- a cross-metric total is not
+ // well defined and is left as future work.)
+ const metricKey = record.__metricKey as unknown as string | undefined;
+ if (metricKey) {
+ const realColCount = levelColumns.filter(c => c !== metricKey).length;
+ const realRowCount = levelRows.filter(r => r !== metricKey).length;
+ if (levelColumns.includes(metricKey) && realColCount === 0) {
+ if (rowKey.length === 0) this.allTotal.push(record);
+ else this.rowTotals[flatRowKey]?.push(record);
}
- for (let ci = colStart; ci <= colKey.length; ci += 1) {
- isColSubtotal = ci < colKey.length;
- const fColKey = colKey.slice(0, ci);
- const flatColKey = flatKey(fColKey);
- if (!this.tree[flatRowKey][flatColKey]) {
- this.tree[flatRowKey][flatColKey] = this.getFormattedAggregator(
- record,
- )(this, fRowKey, fColKey);
- }
- this.tree[flatRowKey][flatColKey].push(record);
-
- this.tree[flatRowKey][flatColKey].isRowSubtotal = isRowSubtotal;
- this.tree[flatRowKey][flatColKey].isColSubtotal = isColSubtotal;
- this.tree[flatRowKey][flatColKey].isSubtotal =
- isRowSubtotal || isColSubtotal;
+ if (levelRows.includes(metricKey) && realRowCount === 0) {
+ if (colKey.length === 0) this.allTotal.push(record);
+ else this.colTotals[flatColKey]?.push(record);
}
Review Comment:
<!-- Bito Reply -->
The reviewer's suggestion to add `this.subtotals.rowEnabled` and `this.subtotals.colEnabled` checks to the metric-collapse logic is appropriate. This ensures that the metric-collapse totals are only mirrored when the corresponding subtotals are enabled, maintaining consistency with the guard pattern used elsewhere in the `processRecord` method.
**superset-frontend/plugins/plugin-chart-pivot-table/src/react-pivottable/utilities.ts**
```
if (levelColumns.includes(metricKey) && realColCount === 0) {
if (rowKey.length === 0) this.allTotal.push(record);
else if (this.subtotals.rowEnabled) this.rowTotals[flatRowKey]?.push(record);
}
if (levelRows.includes(metricKey) && realRowCount === 0) {
if (colKey.length === 0) this.allTotal.push(record);
else if (this.subtotals.colEnabled) this.colTotals[flatColKey]?.push(record);
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(table/pivot-table): correct non-additive totals/subtotals via DB rollup [POC for SIP-216] [superset]
Posted by "bito-code-review[bot] (via GitHub)" <gi...@apache.org>.
bito-code-review[bot] commented on PR #41184:
URL: https://github.com/apache/superset/pull/41184#issuecomment-4949798925
AI Code Review is in progress (usually takes 3 to 15 minutes unless it's a very large PR).
<details>
<summary>
Bito Usage Guide
</summary>
**Commands**
Type the following command in the pull request comment and save the comment.
- `/review` - Manually triggers a full AI review.
- `/pause` - Pauses automatic reviews on this pull request.
- `/resume` - Resumes automatic reviews.
- `/resolve` - Marks all Bito-posted review comments as resolved.
- `/abort` - Cancels all in-progress reviews.
Refer to the <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/available-commands" target="_blank" rel="noopener noreferrer">documentation</a> for additional commands.
**Configuration**
This repository uses `Superset` You can customize the agent settings <a href="https://alpha.bito.ai/home/ai-agents/code-review-agent" target="_blank" rel="noopener noreferrer">here</a> or contact your Bito workspace admin at evan@preset.io.
**Documentation & Help**
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/getting-started/install-run-using-bito-cloud/create-or-customize-an-agent-instance" target="_blank" rel="noopener noreferrer">Customize agent settings</a>
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/implementing-custom-code-review-rules" target="_blank" rel="noopener noreferrer">Review rules</a>
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent" target="_blank" rel="noopener noreferrer">General documentation</a>
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/faqs" target="_blank" rel="noopener noreferrer">FAQ</a>
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(table/pivot-table): correct non-additive totals/subtotals via DB rollup [SIP-216] [superset]
Posted by "rusackas (via GitHub)" <gi...@apache.org>.
rusackas commented on PR #41184:
URL: https://github.com/apache/superset/pull/41184#issuecomment-5049116146
Been trying to get reviews on this for over a month by nagging people on slack, but I'll take the approval (thanks!) and lack of bot or human complaints as good enough... merging during the breaking change window to get it into 7.0, and will happily make touchups/fixes as needed when people do more testing than I already have.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(table/pivot-table): correct non-additive totals/subtotals via DB rollup [SIP-216] [superset]
Posted by "rusackas (via GitHub)" <gi...@apache.org>.
rusackas merged PR #41184:
URL: https://github.com/apache/superset/pull/41184
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(table/pivot-table): correct non-additive totals/subtotals via DB rollup [SIP-216] [superset]
Posted by "jermainebarteauix81-droid (via GitHub)" <gi...@apache.org>.
jermainebarteauix81-droid commented on PR #41184:
URL: https://github.com/apache/superset/pull/41184#issuecomment-5826930677
> ### SUMMARY
>
> Proof-of-concept for **[SIP-216] / #41463** — fixing one of Superset's longest-standing correctness gaps: **totals and subtotals are wrong for non-additive metrics** (ratios, `COUNT_DISTINCT`, `AVG`, percentiles, and "Percentage metrics") in both the Pivot Table and the regular Table chart.
>
> See **#41463** for the full proposal, motivation, and rejected alternatives. This PR is the working implementation referenced throughout that SIP.
>
> **Core principle:** totals are computed by the database at the grouping granularity, never by re-aggregating already-aggregated cells. A SQL-aggregate metric is correct at any grouping level if the same expression is evaluated grouped at that level, so no formula language is needed.
>
> **What's here (consolidates #25747, #32260, #38674, #36165, #37627, #34350/#34425/#34426):**
> - **Pivot Table** — database-computed rollup levels via a single `GROUPING SETS` query (native where the engine supports it, per-level fallback otherwise); the react-pivottable aggregator becomes a passthrough that places DB-computed values. Removes the now-meaningless "Aggregation function" control.
> - **Table chart** — retains post-processing on the summary query so percent/contribution columns are recomputed instead of dropped (#37627/#34350).
> - **Performance** — additive fast-path (single query + client-side synthesis), `supports_grouping_sets` engine capability, query-level pruning to only displayed totals.
>
> Builds on prior art in #34592 (salvages its rollup-combination generator). Out of scope: #39223.
>
> ### BEFORE/AFTER SCREENSHOTS OR ANIMATED GIF
>
> A `rows=[gender, state]` pivot with a ratio metric `SUM(CASE WHEN state='CA' …)/SUM(num)`:
>
> - **Before:** grand total / subtotal show the sum of per-row percentages (e.g. 100%+).
> - **After:** grand total and the per-gender subtotal both show the DB-computed **11%**, returned by a single `GROUPING SETS` query (verified in-app on Postgres; 25 rows = 22 leaf + 2 subtotals + 1 grand total).
>
> ### TESTING INSTRUCTIONS
>
> 1. Pivot a non-additive metric (e.g. a ratio `SUM(a)/SUM(b)` or `COUNT_DISTINCT`) with row totals/subtotals enabled; confirm the totals reflect the metric recomputed at each level, not the sum of cells.
> 2. Table chart with a "Percentage metrics" column + "Show summary"; confirm the percent column appears in the summary row.
> 3. Verify additive metrics (`SUM`/`COUNT`) are unchanged.
>
> Unit + integration tests cover the rollup synthesis, GROUPING SETS clause/marker/split helpers (native + SQLite fallback), additivity detection, and the table summary path.
>
> ### ADDITIONAL INFORMATION
>
> - [x] Has associated issue: #41463 (SIP), #25747 (canonical bug); closes #35089
> - [ ] Required feature flags:
> - [x] Changes UI
> - [ ] Includes DB Migration
> - [ ] Introduces new feature or API
> - [x] Removes existing feature or API <!-- pivot "Aggregation function" control; see UPDATING.md -->
>
> > The SIP (#41463) passed its vote on the dev@ list.
>
>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(table/pivot-table): correct non-additive totals/subtotals via DB rollup [POC for SIP-216] [superset]
Posted by "codeant-ai-for-open-source[bot] (via GitHub)" <gi...@apache.org>.
codeant-ai-for-open-source[bot] commented on code in PR #41184:
URL: https://github.com/apache/superset/pull/41184#discussion_r3565321633
##########
superset-frontend/plugins/plugin-chart-pivot-table/test/plugin/utilities.test.ts:
##########
@@ -0,0 +1,446 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import { TimeGranularity } from '@superset-ui/core';
+import buildGroupbyCombinations, {
+ isAdditiveMetric,
+ allMetricsAdditive,
+ additiveReducerFor,
+ synthesizeAdditiveLevels,
+ splitGroupingSetsResult,
+ groupingMarkerLabel,
+} from '../../src/plugin/utilities';
+import { PivotTableQueryFormData, MetricsLayoutEnum } from '../../src/types';
+
+const baseFormData = {
+ groupbyRows: ['row1', 'row2'],
+ groupbyColumns: ['col1', 'col2'],
+ metrics: ['metric1', 'metric2'],
+ tableRenderer: 'Table With Subtotal',
+ colOrder: 'key_a_to_z',
+ rowOrder: 'key_a_to_z',
+ aggregateFunction: 'Sum',
+ metricsLayout: MetricsLayoutEnum.ROWS,
+ transposePivot: false,
+ rowSubtotalPosition: true,
+ colSubtotalPosition: true,
+ colTotals: true,
+ colSubTotals: true,
+ rowTotals: true,
+ rowSubTotals: true,
+ valueFormat: 'SMART_NUMBER',
+ datasource: '5__table',
+ viz_type: 'my_chart',
+ width: 800,
+ height: 600,
+ combineMetric: false,
+ verboseMap: {},
+ columnFormats: {},
+ currencyFormats: {},
+ metricColorFormatters: [],
+ dateFormatters: {},
+ setDataMask: () => {},
+ legacy_order_by: 'count',
+ order_desc: true,
+ margin: 0,
+ time_grain_sqla: TimeGranularity.MONTH,
+ temporal_columns_lookup: { col1: true },
+ currencyFormat: { symbol: 'USD', symbolPosition: 'prefix' },
+} as unknown as PivotTableQueryFormData;
+
+test('should build all combinations for basic pivot table', () => {
+ const combinations = buildGroupbyCombinations(baseFormData);
+
+ expect(combinations).toEqual([
+ { rows: [], columns: [] },
+ { rows: [], columns: ['col1'] },
+ { rows: [], columns: ['col1', 'col2'] },
+ { rows: ['row1'], columns: [] },
+ { rows: ['row1'], columns: ['col1'] },
+ { rows: ['row1'], columns: ['col1', 'col2'] },
+ { rows: ['row1', 'row2'], columns: [] },
+ { rows: ['row1', 'row2'], columns: ['col1'] },
+ { rows: ['row1', 'row2'], columns: ['col1', 'col2'] },
+ ]);
+
+ expect(combinations).toHaveLength(9);
+});
+
+test('should handle transposed pivot correctly', () => {
+ const modifiedFormData = {
+ ...baseFormData,
+ transposePivot: true,
+ };
+
+ const combinations = buildGroupbyCombinations(modifiedFormData);
+
+ expect(combinations).toEqual([
+ { rows: [], columns: [] },
+ { rows: [], columns: ['row1'] },
+ { rows: [], columns: ['row1', 'row2'] },
+ { rows: ['col1'], columns: [] },
+ { rows: ['col1'], columns: ['row1'] },
+ { rows: ['col1'], columns: ['row1', 'row2'] },
+ { rows: ['col1', 'col2'], columns: [] },
+ { rows: ['col1', 'col2'], columns: ['row1'] },
+ { rows: ['col1', 'col2'], columns: ['row1', 'row2'] },
+ ]);
+});
+
+test('should filter combinations when combineMetric is true with ROWS layout', () => {
+ const modifiedFormData = {
+ ...baseFormData,
+ combineMetric: true,
+ metricsLayout: MetricsLayoutEnum.ROWS,
+ };
+
+ const combinations = buildGroupbyCombinations(modifiedFormData);
+
+ expect(combinations).toEqual([
+ { rows: ['row1', 'row2'], columns: [] },
+ { rows: ['row1', 'row2'], columns: ['col1'] },
+ { rows: ['row1', 'row2'], columns: ['col1', 'col2'] },
+ ]);
+
+ expect(combinations).toHaveLength(3);
+});
+
+test('should filter combinations when combineMetric is true with COLUMNS layout', () => {
+ const modifiedFormData = {
+ ...baseFormData,
+ combineMetric: true,
+ metricsLayout: MetricsLayoutEnum.COLUMNS,
+ };
+
+ const combinations = buildGroupbyCombinations(modifiedFormData);
+
+ expect(combinations).toEqual([
+ { rows: [], columns: ['col1', 'col2'] },
+ { rows: ['row1'], columns: ['col1', 'col2'] },
+ { rows: ['row1', 'row2'], columns: ['col1', 'col2'] },
+ ]);
+
+ expect(combinations).toHaveLength(3);
+});
+
+test('should handle single dimension in rows only', () => {
+ const modifiedFormData = {
+ ...baseFormData,
+ groupbyRows: ['row'],
+ groupbyColumns: [],
+ };
+
+ const combinations = buildGroupbyCombinations(modifiedFormData);
+
+ expect(combinations).toEqual([
+ { rows: [], columns: [] },
+ { rows: ['row'], columns: [] },
+ ]);
+
+ expect(combinations).toHaveLength(2);
+});
+
+test('should handle single dimension in columns only', () => {
+ const modifiedFormData = {
+ ...baseFormData,
+ groupbyRows: [],
+ groupbyColumns: ['col'],
+ };
+
+ const combinations = buildGroupbyCombinations(modifiedFormData);
+
+ expect(combinations).toEqual([
+ { rows: [], columns: [] },
+ { rows: [], columns: ['col'] },
+ ]);
+
+ expect(combinations).toHaveLength(2);
+});
+
+test('should handle empty groupby arrays', () => {
+ const modifiedFormData = {
+ ...baseFormData,
+ groupbyRows: [],
+ groupbyColumns: [],
+ };
+
+ const combinations = buildGroupbyCombinations(modifiedFormData);
+
+ expect(combinations).toEqual([{ rows: [], columns: [] }]);
+
+ expect(combinations).toHaveLength(1);
+});
+
+test('should work with combineMetric and transposed pivot', () => {
+ const modifiedFormData = {
+ ...baseFormData,
+ transposePivot: true,
+ combineMetric: true,
+ metricsLayout: MetricsLayoutEnum.COLUMNS,
+ };
+
+ const combinations = buildGroupbyCombinations(modifiedFormData);
+
+ expect(combinations).toEqual([
+ { rows: [], columns: ['row1', 'row2'] },
+ { rows: ['col1'], columns: ['row1', 'row2'] },
+ { rows: ['col1', 'col2'], columns: ['row1', 'row2'] },
+ ]);
+});
+
+test('should handle combineMetric with empty arrays correctly', () => {
+ const modifiedFormData = {
+ ...baseFormData,
+ groupbyRows: [],
+ groupbyColumns: ['col'],
+ combineMetric: true,
+ metricsLayout: MetricsLayoutEnum.ROWS,
+ };
+
+ const combinations = buildGroupbyCombinations(modifiedFormData);
+
+ expect(combinations).toEqual([
+ { rows: [], columns: [] },
+ { rows: [], columns: ['col'] },
+ ]);
+});
+
+test('should work with large number of dimensions', () => {
+ const modifiedFormData = {
+ ...baseFormData,
+ groupbyRows: ['r1', 'r2', 'r3', 'r4'],
+ groupbyColumns: ['c1', 'c2', 'c3'],
+ };
+
+ const combinations = buildGroupbyCombinations(modifiedFormData);
+
+ expect(combinations).toHaveLength(20);
+
+ expect(combinations).toContainEqual({ rows: [], columns: [] });
+ expect(combinations).toContainEqual({
+ rows: ['r1', 'r2', 'r3', 'r4'],
+ columns: ['c1', 'c2', 'c3'],
+ });
+});
+
+test('isAdditiveMetric: SIMPLE metrics with additive aggregates are additive', () => {
+ expect(
+ isAdditiveMetric({
+ expressionType: 'SIMPLE',
+ aggregate: 'SUM',
+ column: { column_name: 'num' },
+ label: 'sum_num',
+ } as any),
+ ).toBe(true);
+ expect(
+ isAdditiveMetric({
+ expressionType: 'SIMPLE',
+ aggregate: 'COUNT',
+ column: { column_name: 'num' },
+ label: 'count_num',
+ } as any),
+ ).toBe(true);
+});
+
+test('isAdditiveMetric: non-additive aggregates, SQL, and saved metrics are not additive', () => {
+ expect(
+ isAdditiveMetric({
+ expressionType: 'SIMPLE',
+ aggregate: 'AVG',
+ column: { column_name: 'num' },
+ label: 'avg_num',
+ } as any),
+ ).toBe(false);
+ expect(
+ isAdditiveMetric({
+ expressionType: 'SIMPLE',
+ aggregate: 'COUNT_DISTINCT',
+ column: { column_name: 'name' },
+ label: 'distinct_names',
+ } as any),
+ ).toBe(false);
+ expect(
+ isAdditiveMetric({
+ expressionType: 'SQL',
+ sqlExpression: 'SUM(a) / SUM(b)',
+ label: 'ratio',
+ } as any),
+ ).toBe(false);
+ // saved-metric reference: aggregate unknown from form data -> non-additive
+ expect(isAdditiveMetric('count' as any)).toBe(false);
Review Comment:
**Suggestion:** Replace the string `any` cast with a properly typed saved-metric reference value accepted by `isAdditiveMetric`. [custom_rule]
**Severity Level:** Minor 🧹
<details>
<summary><b>Why it matters? ⭐ </b></summary>
The code uses `as any` on a string literal in TypeScript. That is exactly the prohibited pattern under the no-any rule.
</details>
<details>
<summary><b>Rule source 📖 </b></summary>
.cursor/rules/dev-standard.mdc (line 16)
</details>
[](https://app.codeant.ai/fix-in-ide?tool=cursor&prompt_id=b3e1a2aab75f4edd8c301da4eff6aa9c&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset) [](https://app.codeant.ai/fix-in-ide?tool=vscode-claude&prompt_id=b3e1a2aab75f4edd8c301da4eff6aa9c&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset)
*(Use Cmd/Ctrl + Click for best experience)*
<details>
<summary><b>Prompt for AI Agent 🤖 </b></summary>
```mdx
This is a comment left during a code review.
**Path:** superset-frontend/plugins/plugin-chart-pivot-table/test/plugin/utilities.test.ts
**Line:** 285:286
**Comment:**
*Custom Rule: Replace the string `any` cast with a properly typed saved-metric reference value accepted by `isAdditiveMetric`.
Validate the correctness of the flagged issue. If correct, How can I resolve this? If you propose a fix, implement it and please make it concise.
Once fix is implemented, also check other comments on the same PR, and ask user if the user wants to fix the rest of the comments as well. if said yes, then fetch all the comments validate the correctness and implement a minimal fix
```
</details>
<a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F41184&comment_hash=b75a7bee0d9a495c6caa8675cef4d9fd9ab589bdea6d7df4b2a146df9a7516fd&reaction=like'>👍</a> | <a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F41184&comment_hash=b75a7bee0d9a495c6caa8675cef4d9fd9ab589bdea6d7df4b2a146df9a7516fd&reaction=dislike'>👎</a>
##########
superset/common/grouping_sets.py:
##########
@@ -0,0 +1,117 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""
+SQL building blocks for the pivot-table non-additive totals optimization
+(SIP.md, phase 3b). When a datasource engine reports
+``supports_grouping_sets``, the N per-rollup-level queries can be collapsed into
+a single ``GROUPING SETS`` query: the database computes every level in one scan,
+and each returned row is attributed to its level via ``GROUPING()`` markers.
+
+These are the engine-agnostic SQL primitives. Wiring them into the query context
+(emitting one query and splitting the result back into per-level results) is the
+remaining integration; see SIP.md.
+"""
+
+from __future__ import annotations
+
+from collections.abc import Sequence
+from typing import Final
+
+import pandas as pd
+from sqlalchemy import func, tuple_
+from sqlalchemy.sql.elements import ColumnElement
+
+# Suffix for the per-column GROUPING() marker columns added to a GROUPING SETS
+# query. Chosen to be unlikely to collide with a real metric/column label.
+GROUPING_MARKER_SUFFIX: Final = "__superset_grouping"
+
+
+def grouping_marker_label(column_label: str) -> str:
+ """The output label of the GROUPING() marker for a groupby column."""
+ return f"{column_label}{GROUPING_MARKER_SUFFIX}"
+
+
+def grouping_sets_clause(
+ groups: Sequence[Sequence[ColumnElement]],
+) -> ColumnElement:
+ """
+ Build a ``GROUP BY GROUPING SETS (...)`` clause from rollup column groups.
+
+ Each group is the set of columns grouped at one rollup level; the empty
+ group ``()`` is the grand total. For example, groups ``[[a, b], [a], []]``
+ produce ``GROUPING SETS ((a, b), (a), ())``.
+
+ :param groups: one column list per rollup level
+ :return: a clause element suitable for ``select(...).group_by(...)``
+ """
+ return func.grouping_sets(*[tuple_(*group) for group in groups])
+
+
+def grouping_id_column(column: ColumnElement, label: str) -> ColumnElement:
+ """
+ Build a ``GROUPING(col) AS label`` marker column.
+
+ In a ``GROUPING SETS`` result, ``GROUPING(col)`` is ``0`` when ``col`` is
+ part of the row's grouping level and ``1`` when it has been rolled up
+ (aggregated away). Selecting one marker per groupby column lets the caller
+ attribute each returned row to its rollup level when splitting the single
+ result back into per-level results.
+
+ :param column: the groupby column to probe
+ :param label: the output label for the marker (see ``grouping_marker_label``)
+ :return: the labelled ``GROUPING(col)`` column
+ """
+ return func.grouping(column).label(label)
+
+
+def split_grouping_sets_result(
+ df: pd.DataFrame,
+ levels: Sequence[Sequence[str]],
+ groupby_columns: Sequence[str],
+) -> list[pd.DataFrame]:
+ """
+ Split a combined ``GROUPING SETS`` result into one DataFrame per rollup
+ level, the inverse of {@link grouping_sets_clause}.
+
+ Each row of ``df`` carries a ``GROUPING()`` marker per groupby column (named
+ by ``grouping_marker_label``): ``0`` if the column is grouped at that row's
+ level, ``1`` if it was rolled up. A row belongs to a level iff its markers
+ are ``0`` exactly for that level's columns. Marker columns are dropped from
+ the returned frames so each looks like an ordinary per-level query result.
+
+ :param df: the combined query result, including marker columns
+ :param levels: the grouped-column list for each rollup level (same order as
+ passed to ``grouping_sets_clause``)
+ :param groupby_columns: every groupby column that has a marker
+ :return: one DataFrame per level, in ``levels`` order
+ """
+ markers = [grouping_marker_label(col) for col in groupby_columns]
+ results: list[pd.DataFrame] = []
+ for level in levels:
+ grouped = set(level)
+ mask = pd.Series(True, index=df.index)
Review Comment:
**Suggestion:** Add a type annotation for this boolean mask series to keep local data-shaping variables explicitly typed under the custom rule. [custom_rule]
**Severity Level:** Minor 🧹
<details>
<summary><b>Why it matters? ⭐ </b></summary>
This newly introduced local variable is also plainly annotatable, and the code omits a type hint, so the suggestion accurately identifies a rule violation.
</details>
<details>
<summary><b>Rule source 📖 </b></summary>
.cursor/rules/dev-standard.mdc (line 28)
</details>
[](https://app.codeant.ai/fix-in-ide?tool=cursor&prompt_id=1836d796f234402cae413e37ab3a8128&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset) [](https://app.codeant.ai/fix-in-ide?tool=vscode-claude&prompt_id=1836d796f234402cae413e37ab3a8128&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset)
*(Use Cmd/Ctrl + Click for best experience)*
<details>
<summary><b>Prompt for AI Agent 🤖 </b></summary>
```mdx
This is a comment left during a code review.
**Path:** superset/common/grouping_sets.py
**Line:** 107:107
**Comment:**
*Custom Rule: Add a type annotation for this boolean mask series to keep local data-shaping variables explicitly typed under the custom rule.
Validate the correctness of the flagged issue. If correct, How can I resolve this? If you propose a fix, implement it and please make it concise.
Once fix is implemented, also check other comments on the same PR, and ask user if the user wants to fix the rest of the comments as well. if said yes, then fetch all the comments validate the correctness and implement a minimal fix
```
</details>
<a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F41184&comment_hash=5c3e55d3e7dc707fe21d805618879b34549d15d62b12c83b417bddfeb436f099&reaction=like'>👍</a> | <a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F41184&comment_hash=5c3e55d3e7dc707fe21d805618879b34549d15d62b12c83b417bddfeb436f099&reaction=dislike'>👎</a>
##########
superset/common/grouping_sets.py:
##########
@@ -0,0 +1,117 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""
+SQL building blocks for the pivot-table non-additive totals optimization
+(SIP.md, phase 3b). When a datasource engine reports
+``supports_grouping_sets``, the N per-rollup-level queries can be collapsed into
+a single ``GROUPING SETS`` query: the database computes every level in one scan,
+and each returned row is attributed to its level via ``GROUPING()`` markers.
+
+These are the engine-agnostic SQL primitives. Wiring them into the query context
+(emitting one query and splitting the result back into per-level results) is the
+remaining integration; see SIP.md.
+"""
+
+from __future__ import annotations
+
+from collections.abc import Sequence
+from typing import Final
+
+import pandas as pd
+from sqlalchemy import func, tuple_
+from sqlalchemy.sql.elements import ColumnElement
+
+# Suffix for the per-column GROUPING() marker columns added to a GROUPING SETS
+# query. Chosen to be unlikely to collide with a real metric/column label.
+GROUPING_MARKER_SUFFIX: Final = "__superset_grouping"
+
+
+def grouping_marker_label(column_label: str) -> str:
+ """The output label of the GROUPING() marker for a groupby column."""
+ return f"{column_label}{GROUPING_MARKER_SUFFIX}"
+
+
+def grouping_sets_clause(
+ groups: Sequence[Sequence[ColumnElement]],
+) -> ColumnElement:
+ """
+ Build a ``GROUP BY GROUPING SETS (...)`` clause from rollup column groups.
+
+ Each group is the set of columns grouped at one rollup level; the empty
+ group ``()`` is the grand total. For example, groups ``[[a, b], [a], []]``
+ produce ``GROUPING SETS ((a, b), (a), ())``.
+
+ :param groups: one column list per rollup level
+ :return: a clause element suitable for ``select(...).group_by(...)``
+ """
+ return func.grouping_sets(*[tuple_(*group) for group in groups])
+
+
+def grouping_id_column(column: ColumnElement, label: str) -> ColumnElement:
+ """
+ Build a ``GROUPING(col) AS label`` marker column.
+
+ In a ``GROUPING SETS`` result, ``GROUPING(col)`` is ``0`` when ``col`` is
+ part of the row's grouping level and ``1`` when it has been rolled up
+ (aggregated away). Selecting one marker per groupby column lets the caller
+ attribute each returned row to its rollup level when splitting the single
+ result back into per-level results.
+
+ :param column: the groupby column to probe
+ :param label: the output label for the marker (see ``grouping_marker_label``)
+ :return: the labelled ``GROUPING(col)`` column
+ """
+ return func.grouping(column).label(label)
+
+
+def split_grouping_sets_result(
+ df: pd.DataFrame,
+ levels: Sequence[Sequence[str]],
+ groupby_columns: Sequence[str],
+) -> list[pd.DataFrame]:
+ """
+ Split a combined ``GROUPING SETS`` result into one DataFrame per rollup
+ level, the inverse of {@link grouping_sets_clause}.
+
+ Each row of ``df`` carries a ``GROUPING()`` marker per groupby column (named
+ by ``grouping_marker_label``): ``0`` if the column is grouped at that row's
+ level, ``1`` if it was rolled up. A row belongs to a level iff its markers
+ are ``0`` exactly for that level's columns. Marker columns are dropped from
+ the returned frames so each looks like an ordinary per-level query result.
+
+ :param df: the combined query result, including marker columns
+ :param levels: the grouped-column list for each rollup level (same order as
+ passed to ``grouping_sets_clause``)
+ :param groupby_columns: every groupby column that has a marker
+ :return: one DataFrame per level, in ``levels`` order
+ """
+ markers = [grouping_marker_label(col) for col in groupby_columns]
Review Comment:
**Suggestion:** Add an explicit type annotation for this derived collection variable to satisfy the rule requiring type hints on relevant annotatable variables. [custom_rule]
**Severity Level:** Minor 🧹
<details>
<summary><b>Why it matters? ⭐ </b></summary>
This new local variable is a clearly annotatable list and is left untyped in newly added Python code, which matches the custom rule requiring type hints on relevant variables.
</details>
<details>
<summary><b>Rule source 📖 </b></summary>
.cursor/rules/dev-standard.mdc (line 28)
</details>
[](https://app.codeant.ai/fix-in-ide?tool=cursor&prompt_id=b3119cd55c034cb5b88be461e58bc0ac&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset) [](https://app.codeant.ai/fix-in-ide?tool=vscode-claude&prompt_id=b3119cd55c034cb5b88be461e58bc0ac&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset)
*(Use Cmd/Ctrl + Click for best experience)*
<details>
<summary><b>Prompt for AI Agent 🤖 </b></summary>
```mdx
This is a comment left during a code review.
**Path:** superset/common/grouping_sets.py
**Line:** 103:103
**Comment:**
*Custom Rule: Add an explicit type annotation for this derived collection variable to satisfy the rule requiring type hints on relevant annotatable variables.
Validate the correctness of the flagged issue. If correct, How can I resolve this? If you propose a fix, implement it and please make it concise.
Once fix is implemented, also check other comments on the same PR, and ask user if the user wants to fix the rest of the comments as well. if said yes, then fetch all the comments validate the correctness and implement a minimal fix
```
</details>
<a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F41184&comment_hash=b494a9099c22f5fc19f1148826c70b44305393c84c01a7422c5b1599a920cd74&reaction=like'>👍</a> | <a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F41184&comment_hash=b494a9099c22f5fc19f1148826c70b44305393c84c01a7422c5b1599a920cd74&reaction=dislike'>👎</a>
##########
superset/common/grouping_sets.py:
##########
@@ -0,0 +1,117 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""
+SQL building blocks for the pivot-table non-additive totals optimization
+(SIP.md, phase 3b). When a datasource engine reports
+``supports_grouping_sets``, the N per-rollup-level queries can be collapsed into
+a single ``GROUPING SETS`` query: the database computes every level in one scan,
+and each returned row is attributed to its level via ``GROUPING()`` markers.
+
+These are the engine-agnostic SQL primitives. Wiring them into the query context
+(emitting one query and splitting the result back into per-level results) is the
+remaining integration; see SIP.md.
+"""
+
+from __future__ import annotations
+
+from collections.abc import Sequence
+from typing import Final
+
+import pandas as pd
+from sqlalchemy import func, tuple_
+from sqlalchemy.sql.elements import ColumnElement
+
+# Suffix for the per-column GROUPING() marker columns added to a GROUPING SETS
+# query. Chosen to be unlikely to collide with a real metric/column label.
+GROUPING_MARKER_SUFFIX: Final = "__superset_grouping"
+
+
+def grouping_marker_label(column_label: str) -> str:
+ """The output label of the GROUPING() marker for a groupby column."""
+ return f"{column_label}{GROUPING_MARKER_SUFFIX}"
+
+
+def grouping_sets_clause(
+ groups: Sequence[Sequence[ColumnElement]],
+) -> ColumnElement:
+ """
+ Build a ``GROUP BY GROUPING SETS (...)`` clause from rollup column groups.
+
+ Each group is the set of columns grouped at one rollup level; the empty
+ group ``()`` is the grand total. For example, groups ``[[a, b], [a], []]``
+ produce ``GROUPING SETS ((a, b), (a), ())``.
+
+ :param groups: one column list per rollup level
+ :return: a clause element suitable for ``select(...).group_by(...)``
+ """
+ return func.grouping_sets(*[tuple_(*group) for group in groups])
+
+
+def grouping_id_column(column: ColumnElement, label: str) -> ColumnElement:
+ """
+ Build a ``GROUPING(col) AS label`` marker column.
+
+ In a ``GROUPING SETS`` result, ``GROUPING(col)`` is ``0`` when ``col`` is
+ part of the row's grouping level and ``1`` when it has been rolled up
+ (aggregated away). Selecting one marker per groupby column lets the caller
+ attribute each returned row to its rollup level when splitting the single
+ result back into per-level results.
+
+ :param column: the groupby column to probe
+ :param label: the output label for the marker (see ``grouping_marker_label``)
+ :return: the labelled ``GROUPING(col)`` column
+ """
+ return func.grouping(column).label(label)
+
+
+def split_grouping_sets_result(
+ df: pd.DataFrame,
+ levels: Sequence[Sequence[str]],
+ groupby_columns: Sequence[str],
+) -> list[pd.DataFrame]:
+ """
+ Split a combined ``GROUPING SETS`` result into one DataFrame per rollup
+ level, the inverse of {@link grouping_sets_clause}.
+
+ Each row of ``df`` carries a ``GROUPING()`` marker per groupby column (named
+ by ``grouping_marker_label``): ``0`` if the column is grouped at that row's
+ level, ``1`` if it was rolled up. A row belongs to a level iff its markers
+ are ``0`` exactly for that level's columns. Marker columns are dropped from
+ the returned frames so each looks like an ordinary per-level query result.
+
+ :param df: the combined query result, including marker columns
+ :param levels: the grouped-column list for each rollup level (same order as
+ passed to ``grouping_sets_clause``)
+ :param groupby_columns: every groupby column that has a marker
+ :return: one DataFrame per level, in ``levels`` order
+ """
+ markers = [grouping_marker_label(col) for col in groupby_columns]
+ results: list[pd.DataFrame] = []
+ for level in levels:
+ grouped = set(level)
Review Comment:
**Suggestion:** Annotate this set variable with its element type so the new code consistently applies type hints to relevant local variables. [custom_rule]
**Severity Level:** Minor 🧹
<details>
<summary><b>Why it matters? ⭐ </b></summary>
This is a new local variable whose type can be annotated, but no type hint is present. That is a real instance of the type-hint rule violation.
</details>
<details>
<summary><b>Rule source 📖 </b></summary>
.cursor/rules/dev-standard.mdc (line 28)
</details>
[](https://app.codeant.ai/fix-in-ide?tool=cursor&prompt_id=d8ef087e6d994f319f554aaa5c8aba44&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset) [](https://app.codeant.ai/fix-in-ide?tool=vscode-claude&prompt_id=d8ef087e6d994f319f554aaa5c8aba44&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset)
*(Use Cmd/Ctrl + Click for best experience)*
<details>
<summary><b>Prompt for AI Agent 🤖 </b></summary>
```mdx
This is a comment left during a code review.
**Path:** superset/common/grouping_sets.py
**Line:** 106:106
**Comment:**
*Custom Rule: Annotate this set variable with its element type so the new code consistently applies type hints to relevant local variables.
Validate the correctness of the flagged issue. If correct, How can I resolve this? If you propose a fix, implement it and please make it concise.
Once fix is implemented, also check other comments on the same PR, and ask user if the user wants to fix the rest of the comments as well. if said yes, then fetch all the comments validate the correctness and implement a minimal fix
```
</details>
<a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F41184&comment_hash=c732a3404d472185458224a53f0758d610add23183c65b12849526255a0cf83f&reaction=like'>👍</a> | <a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F41184&comment_hash=c732a3404d472185458224a53f0758d610add23183c65b12849526255a0cf83f&reaction=dislike'>👎</a>
##########
superset/common/query_context_processor.py:
##########
@@ -252,9 +256,60 @@ def get_query_result(self, query_object: QueryObject) -> QueryResult:
This method delegates to the datasource's get_query_result method,
which handles query execution, normalization, time offsets, and
post-processing.
+
+ When the query requests rollup ``grouping_sets`` but the engine does not
+ support native ``GROUPING SETS``, fall back to one query per level and
+ concatenate the results with ``GROUPING()``-equivalent markers, so the
+ combined result matches the shape the native path produces (SIP.md,
+ phase 3b). Engines that support it run the single native query.
"""
+ if query_object.grouping_sets and not self._supports_grouping_sets():
+ return self._grouping_sets_fallback(query_object)
return self._qc_datasource.get_query_result(query_object)
+ def _supports_grouping_sets(self) -> bool:
+ engine_spec: BaseEngineSpec | None = getattr(
+ self._qc_datasource, "db_engine_spec", None
+ )
+ return bool(engine_spec and engine_spec.supports_grouping_sets)
+
+ def _grouping_sets_fallback(self, query_object: QueryObject) -> QueryResult:
+ """
+ Emulate a GROUPING SETS query on engines without native support: run one
+ query per rollup level and concatenate, tagging each level's rows with
+ the same per-column markers the native path emits.
+ """
+ levels: list[list[str]] = query_object.grouping_sets
+ # Use the same label derivation as the native path (physical column name
+ # or adhoc column label) so both column kinds are represented and each
+ # label maps back to its own column, in the same order as the source
+ # list.
+ all_labels: list[str] = [get_column_name(col) for col in query_object.columns]
+ label_to_column = dict(zip(all_labels, query_object.columns, strict=True))
+
+ frames: list[pd.DataFrame] = []
+ result: QueryResult | None = None
+ for level in levels:
+ level_labels = set(level)
Review Comment:
**Suggestion:** Add an explicit type annotation for this set variable to keep new local variables consistently typed. [custom_rule]
**Severity Level:** Minor 🧹
<details>
<summary><b>Why it matters? ⭐ </b></summary>
This is a newly added local variable in Python code, and it can be explicitly annotated for static typing. The lack of a type hint matches the custom rule.
</details>
<details>
<summary><b>Rule source 📖 </b></summary>
.cursor/rules/dev-standard.mdc (line 28)
</details>
[](https://app.codeant.ai/fix-in-ide?tool=cursor&prompt_id=0aafdd509e8d4d2998fdbe0d89d284aa&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset) [](https://app.codeant.ai/fix-in-ide?tool=vscode-claude&prompt_id=0aafdd509e8d4d2998fdbe0d89d284aa&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset)
*(Use Cmd/Ctrl + Click for best experience)*
<details>
<summary><b>Prompt for AI Agent 🤖 </b></summary>
```mdx
This is a comment left during a code review.
**Path:** superset/common/query_context_processor.py
**Line:** 293:293
**Comment:**
*Custom Rule: Add an explicit type annotation for this set variable to keep new local variables consistently typed.
Validate the correctness of the flagged issue. If correct, How can I resolve this? If you propose a fix, implement it and please make it concise.
Once fix is implemented, also check other comments on the same PR, and ask user if the user wants to fix the rest of the comments as well. if said yes, then fetch all the comments validate the correctness and implement a minimal fix
```
</details>
<a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F41184&comment_hash=2883dd409d4bb1414225604e213d6a922e3b7e1e5bf9c0f7e516c71843b00bbe&reaction=like'>👍</a> | <a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F41184&comment_hash=2883dd409d4bb1414225604e213d6a922e3b7e1e5bf9c0f7e516c71843b00bbe&reaction=dislike'>👎</a>
##########
superset/common/query_context_processor.py:
##########
@@ -252,9 +256,60 @@ def get_query_result(self, query_object: QueryObject) -> QueryResult:
This method delegates to the datasource's get_query_result method,
which handles query execution, normalization, time offsets, and
post-processing.
+
+ When the query requests rollup ``grouping_sets`` but the engine does not
+ support native ``GROUPING SETS``, fall back to one query per level and
+ concatenate the results with ``GROUPING()``-equivalent markers, so the
+ combined result matches the shape the native path produces (SIP.md,
+ phase 3b). Engines that support it run the single native query.
"""
+ if query_object.grouping_sets and not self._supports_grouping_sets():
+ return self._grouping_sets_fallback(query_object)
return self._qc_datasource.get_query_result(query_object)
+ def _supports_grouping_sets(self) -> bool:
+ engine_spec: BaseEngineSpec | None = getattr(
+ self._qc_datasource, "db_engine_spec", None
+ )
+ return bool(engine_spec and engine_spec.supports_grouping_sets)
+
+ def _grouping_sets_fallback(self, query_object: QueryObject) -> QueryResult:
+ """
+ Emulate a GROUPING SETS query on engines without native support: run one
+ query per rollup level and concatenate, tagging each level's rows with
+ the same per-column markers the native path emits.
+ """
+ levels: list[list[str]] = query_object.grouping_sets
+ # Use the same label derivation as the native path (physical column name
+ # or adhoc column label) so both column kinds are represented and each
+ # label maps back to its own column, in the same order as the source
+ # list.
+ all_labels: list[str] = [get_column_name(col) for col in query_object.columns]
+ label_to_column = dict(zip(all_labels, query_object.columns, strict=True))
Review Comment:
**Suggestion:** Add an explicit type annotation for this mapping so static type checking can validate key/value usage. [custom_rule]
**Severity Level:** Minor 🧹
<details>
<summary><b>Why it matters? ⭐ </b></summary>
This is newly introduced Python code in a modified file, and the local mapping variable can be annotated to satisfy the type-hint rule. The suggestion correctly identifies an omitted type hint on an annotatable variable.
</details>
<details>
<summary><b>Rule source 📖 </b></summary>
.cursor/rules/dev-standard.mdc (line 28)
</details>
[](https://app.codeant.ai/fix-in-ide?tool=cursor&prompt_id=6e481d46188046cead535dd509b9c49d&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset) [](https://app.codeant.ai/fix-in-ide?tool=vscode-claude&prompt_id=6e481d46188046cead535dd509b9c49d&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset)
*(Use Cmd/Ctrl + Click for best experience)*
<details>
<summary><b>Prompt for AI Agent 🤖 </b></summary>
```mdx
This is a comment left during a code review.
**Path:** superset/common/query_context_processor.py
**Line:** 288:288
**Comment:**
*Custom Rule: Add an explicit type annotation for this mapping so static type checking can validate key/value usage.
Validate the correctness of the flagged issue. If correct, How can I resolve this? If you propose a fix, implement it and please make it concise.
Once fix is implemented, also check other comments on the same PR, and ask user if the user wants to fix the rest of the comments as well. if said yes, then fetch all the comments validate the correctness and implement a minimal fix
```
</details>
<a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F41184&comment_hash=f07d2a8670d1f33d82e898aac8a182938924a5c7d3220bf2ac71a2588b2b9078&reaction=like'>👍</a> | <a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F41184&comment_hash=f07d2a8670d1f33d82e898aac8a182938924a5c7d3220bf2ac71a2588b2b9078&reaction=dislike'>👎</a>
##########
superset-frontend/plugins/plugin-chart-pivot-table/test/plugin/transformProps.test.ts:
##########
@@ -64,36 +63,73 @@ const chartProps = new ChartProps<QueryFormData>({
theme: supersetTheme,
Review Comment:
**Suggestion:** Replace this `any` cast with the concrete return type of `transformProps` so the test remains type-safe. [custom_rule]
**Severity Level:** Minor 🧹
<details>
<summary><b>Why it matters? ⭐ </b></summary>
The code explicitly casts the transform result to `any`, which matches the custom rule violation for TypeScript code using `any` instead of a concrete type.
</details>
<details>
<summary><b>Rule source 📖 </b></summary>
.cursor/rules/dev-standard.mdc (line 16)
</details>
[](https://app.codeant.ai/fix-in-ide?tool=cursor&prompt_id=65f58aaea0724244a242e07d3ee33669&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset) [](https://app.codeant.ai/fix-in-ide?tool=vscode-claude&prompt_id=65f58aaea0724244a242e07d3ee33669&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset)
*(Use Cmd/Ctrl + Click for best experience)*
<details>
<summary><b>Prompt for AI Agent 🤖 </b></summary>
```mdx
This is a comment left during a code review.
**Path:** superset-frontend/plugins/plugin-chart-pivot-table/test/plugin/transformProps.test.ts
**Line:** 63:63
**Comment:**
*Custom Rule: Replace this `any` cast with the concrete return type of `transformProps` so the test remains type-safe.
Validate the correctness of the flagged issue. If correct, How can I resolve this? If you propose a fix, implement it and please make it concise.
Once fix is implemented, also check other comments on the same PR, and ask user if the user wants to fix the rest of the comments as well. if said yes, then fetch all the comments validate the correctness and implement a minimal fix
```
</details>
<a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F41184&comment_hash=f694e50b82c66df01be21ea5cee8a1aebec8c71c48d0be1ce2b6c5736ef0de7a&reaction=like'>👍</a> | <a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F41184&comment_hash=f694e50b82c66df01be21ea5cee8a1aebec8c71c48d0be1ce2b6c5736ef0de7a&reaction=dislike'>👎</a>
##########
superset-frontend/plugins/plugin-chart-pivot-table/test/plugin/transformProps.test.ts:
##########
@@ -64,36 +63,73 @@ const chartProps = new ChartProps<QueryFormData>({
theme: supersetTheme,
});
-test('should transform chart props for viz', () => {
- expect(transformProps(chartProps)).toEqual({
- width: 800,
- height: 600,
- groupbyRows: ['row1', 'row2'],
- groupbyColumns: ['col1', 'col2'],
- metrics: ['metric1', 'metric2'],
- tableRenderer: 'Table With Subtotal',
- colOrder: 'key_a_to_z',
- rowOrder: 'key_a_to_z',
- aggregateFunction: 'Sum',
- transposePivot: true,
- combineMetric: true,
- rowSubtotalPosition: true,
- colSubtotalPosition: true,
+test('should pass through formData props for viz', () => {
+ const result = transformProps(chartProps) as any;
+ expect(result.width).toBe(800);
+ expect(result.height).toBe(600);
+ expect(result.groupbyRows).toEqual(['row1', 'row2']);
+ expect(result.groupbyColumns).toEqual(['col1', 'col2']);
+ expect(result.metrics).toEqual(['metric1', 'metric2']);
+ expect(result.metricsLayout).toBe(MetricsLayoutEnum.COLUMNS);
+ expect(result.currencyFormat).toEqual({
Review Comment:
**Suggestion:** Type this callback parameter with the concrete item shape instead of `any`. [custom_rule]
**Severity Level:** Minor 🧹
<details>
<summary><b>Why it matters? ⭐ </b></summary>
The callback parameter is typed as `any`, which is directly prohibited by the no-any-types rule.
</details>
<details>
<summary><b>Rule source 📖 </b></summary>
.cursor/rules/dev-standard.mdc (line 16)
</details>
[](https://app.codeant.ai/fix-in-ide?tool=cursor&prompt_id=1158f6d6540a475689e952c355b01665&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset) [](https://app.codeant.ai/fix-in-ide?tool=vscode-claude&prompt_id=1158f6d6540a475689e952c355b01665&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset)
*(Use Cmd/Ctrl + Click for best experience)*
<details>
<summary><b>Prompt for AI Agent 🤖 </b></summary>
```mdx
This is a comment left during a code review.
**Path:** superset-frontend/plugins/plugin-chart-pivot-table/test/plugin/transformProps.test.ts
**Line:** 74:74
**Comment:**
*Custom Rule: Type this callback parameter with the concrete item shape instead of `any`.
Validate the correctness of the flagged issue. If correct, How can I resolve this? If you propose a fix, implement it and please make it concise.
Once fix is implemented, also check other comments on the same PR, and ask user if the user wants to fix the rest of the comments as well. if said yes, then fetch all the comments validate the correctness and implement a minimal fix
```
</details>
<a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F41184&comment_hash=881dd6afee054f246ccbc4ba797b4d8a5524ddecf15948f948daeb807018dfee&reaction=like'>👍</a> | <a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F41184&comment_hash=881dd6afee054f246ccbc4ba797b4d8a5524ddecf15948f948daeb807018dfee&reaction=dislike'>👎</a>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(table/pivot-table): correct non-additive totals/subtotals via DB rollup [POC for SIP-216] [superset]
Posted by "codeant-ai-for-open-source[bot] (via GitHub)" <gi...@apache.org>.
codeant-ai-for-open-source[bot] commented on code in PR #41184:
URL: https://github.com/apache/superset/pull/41184#discussion_r3565327481
##########
superset-frontend/plugins/plugin-chart-pivot-table/src/plugin/buildQuery.ts:
##########
@@ -54,6 +59,39 @@ export default function buildQuery(formData: PivotTableQueryFormData) {
}
return col;
});
+}
+
+export default function buildQuery(formData: PivotTableQueryFormData) {
+ const { extra_form_data } = formData;
+ const time_grain_sqla =
+ extra_form_data?.time_grain_sqla || formData.time_grain_sqla;
+
+ // The full set of dimensions (the leaf level) is always queried; the rollup
+ // levels are derived from it. Apply transposePivot here exactly as
+ // buildGroupbyCombinations does, so the column order matches the rollup
+ // levels. See SIP.md.
+ const [fullRows, fullColumnDims] = formData.transposePivot
+ ? [formData.groupbyColumns, formData.groupbyRows]
+ : [formData.groupbyRows, formData.groupbyColumns];
+ const fullGroupby: Groupby = {
+ rows: ensureIsArray<QueryFormColumn>(fullRows),
+ columns: ensureIsArray<QueryFormColumn>(fullColumnDims),
+ };
+ const fullColumns = getQueryColumns(fullGroupby, formData, time_grain_sqla);
+
+ // A single query always suffices:
+ // - additive metrics (SUM/COUNT/MIN/MAX): a full-detail query whose
+ // subtotals/grand totals transformProps derives by client-side reduction.
+ // - non-additive metrics: a GROUPING SETS query (one `grouping_sets` level
+ // per displayed total/subtotal) so the database computes every level; the
+ // backend falls back to per-level queries on engines without native
+ // support. transformProps splits the combined result by level.
+ const additive = allMetricsAdditive(ensureIsArray(formData.metrics));
+ const groupingSets = additive
+ ? undefined
+ : buildGroupbyCombinations(formData).map(level =>
+ [...level.rows, ...level.columns].map(getColumnLabel),
Review Comment:
**Suggestion:** The generated `grouping_sets` levels are not de-duplicated, so if the same dimension is placed on both axes the same column label can appear twice in one level. That propagates duplicate expressions into backend GROUPING SETS generation and can produce invalid SQL or engine-specific failures. De-duplicate labels per level before sending them. [logic error]
<details>
<summary><b>Severity Level:</b> Critical 🚨</summary>
```mdx
❌ Pivot Table queries can fail when axis dimensions overlap.
⚠️ Backend may generate invalid GROUPING SETS SQL.
⚠️ Non-additive rollups brittle for shared row/column dimensions.
```
</details>
<details>
<summary><b>Steps of Reproduction ✅ </b></summary>
```mdx
1. Create a Pivot Table chart where the same dimension (e.g. `state`) is assigned on both
axes so `formData.groupbyRows` and `formData.groupbyColumns` each contain an identical
`QueryFormColumn` entry; these arrays are used by `buildGroupbyCombinations(formData)` in
`superset-frontend/plugins/plugin-chart-pivot-table/src/plugin/utilities.ts:183-231`
without cross-axis deduplication.
2. With any totals/subtotals toggles enabled (`colTotals`, `rowTotals`, `rowSubTotals`, or
`colSubTotals` true), `buildGroupbyCombinations` generates multiple rollup levels; for
levels where the prefixes of `rows` and `columns` both include the shared dimension, the
resulting `Groupby` combination has that dimension in both `rows` and `columns`.
3. In `buildQuery` at
`superset-frontend/plugins/plugin-chart-pivot-table/src/plugin/buildQuery.ts:89-94`,
`groupingSets` is computed as `buildGroupbyCombinations(formData).map(level =>
[...level.rows, ...level.columns].map(getColumnLabel))`. When a dimension appears on both
axes, `[...]` concatenation produces per-level label lists containing the same column
label multiple times (e.g. `['state', 'state', 'gender']`), and there is no per-level
deduplication before attaching `grouping_sets` to the query object.
4. On the backend, `SqlaTable.query` in `superset/models/helpers.py:3550-45` builds native
GROUPING SETS when `grouping_sets` is set and `db_engine_spec.supports_grouping_sets`; it
translates each level label list into SQL columns via `gs_levels =
[[groupby_all_columns[col] for col in level if col in groupby_all_columns] for level in
grouping_sets or []]` and then `qry = qry.group_by(grouping_sets_clause(gs_levels))`.
Because per-level labels are not deduped, `grouping_sets_clause` in
`superset/common/grouping_sets.py:14-27` receives groups where the same `ColumnElement`
appears multiple times in one tuple, yielding GROUPING SETS clauses with repeated columns
that can be syntactically invalid or cause engine-specific failures when executing the
chart query.
```
</details>
[](https://app.codeant.ai/fix-in-ide?tool=cursor&prompt_id=8ed171bb7f93409e9aa72532241a757f&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset) [](https://app.codeant.ai/fix-in-ide?tool=vscode-claude&prompt_id=8ed171bb7f93409e9aa72532241a757f&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset)
*(Use Cmd/Ctrl + Click for best experience)*
<details>
<summary><b>Prompt for AI Agent 🤖 </b></summary>
```mdx
This is a comment left during a code review.
**Path:** superset-frontend/plugins/plugin-chart-pivot-table/src/plugin/buildQuery.ts
**Line:** 93:93
**Comment:**
*Logic Error: The generated `grouping_sets` levels are not de-duplicated, so if the same dimension is placed on both axes the same column label can appear twice in one level. That propagates duplicate expressions into backend GROUPING SETS generation and can produce invalid SQL or engine-specific failures. De-duplicate labels per level before sending them.
Validate the correctness of the flagged issue. If correct, How can I resolve this? If you propose a fix, implement it and please make it concise.
Once fix is implemented, also check other comments on the same PR, and ask user if the user wants to fix the rest of the comments as well. if said yes, then fetch all the comments validate the correctness and implement a minimal fix
```
</details>
<a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F41184&comment_hash=0784cf33ea677289a4495e7e0e4ea0136f9101bc31ad790aa9a5064a6fff4fdd&reaction=like'>👍</a> | <a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F41184&comment_hash=0784cf33ea677289a4495e7e0e4ea0136f9101bc31ad790aa9a5064a6fff4fdd&reaction=dislike'>👎</a>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(table/pivot-table): correct non-additive totals/subtotals via DB rollup [POC for SIP-216] [superset]
Posted by "bito-code-review[bot] (via GitHub)" <gi...@apache.org>.
bito-code-review[bot] commented on PR #41184:
URL: https://github.com/apache/superset/pull/41184#issuecomment-4932028195
AI Code Review is in progress (usually takes 3 to 15 minutes unless it's a very large PR).
<details>
<summary>
Bito Usage Guide
</summary>
**Commands**
Type the following command in the pull request comment and save the comment.
- `/review` - Manually triggers a full AI review.
- `/pause` - Pauses automatic reviews on this pull request.
- `/resume` - Resumes automatic reviews.
- `/resolve` - Marks all Bito-posted review comments as resolved.
- `/abort` - Cancels all in-progress reviews.
Refer to the <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/available-commands" target="_blank" rel="noopener noreferrer">documentation</a> for additional commands.
**Configuration**
This repository uses `Superset` You can customize the agent settings <a href="https://alpha.bito.ai/home/ai-agents/code-review-agent" target="_blank" rel="noopener noreferrer">here</a> or contact your Bito workspace admin at evan@preset.io.
**Documentation & Help**
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/getting-started/install-run-using-bito-cloud/create-or-customize-an-agent-instance" target="_blank" rel="noopener noreferrer">Customize agent settings</a>
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/implementing-custom-code-review-rules" target="_blank" rel="noopener noreferrer">Review rules</a>
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent" target="_blank" rel="noopener noreferrer">General documentation</a>
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/faqs" target="_blank" rel="noopener noreferrer">FAQ</a>
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(table/pivot-table): correct non-additive totals/subtotals via DB rollup [POC for SIP-216] [superset]
Posted by "rusackas (via GitHub)" <gi...@apache.org>.
rusackas commented on code in PR #41184:
URL: https://github.com/apache/superset/pull/41184#discussion_r3557746062
##########
superset-frontend/plugins/plugin-chart-pivot-table/test/react-pivottable/tableRenders.test.tsx:
##########
@@ -103,35 +150,41 @@ test('TableRenderer renders row headers from pivot data', () => {
});
test('TableRenderer renders aggregated cell values', () => {
- const props = buildDefaultProps();
+ const props = buildDefaultProps({
+ data: TAGGED_COUNT_DATA,
+ vals: ['value'],
+ });
renderWithTheme(<TableRenderer {...props} />);
- // With "Count" aggregator, each cell (row x col intersection) should
- // contain "1" because each combination appears exactly once.
+ // Each leaf cell (row x col intersection) holds the DB-computed value "1".
const cells = screen.getAllByRole('gridcell');
const cellTexts = cells.map(cell => cell.textContent);
- // There should be cell values of "1" for each of the four intersections
- // (blue+circle, blue+square, red+circle, red+square).
- const onesCount = cellTexts.filter(text => text === '1').length;
+ // There should be a "1" leaf cell for each of the four intersections
+ // (blue+circle, blue+square, red+circle, red+square). The default formatter
+ // renders with two decimals in this test (production applies the metric's
+ // own format).
+ const onesCount = cellTexts.filter(text => text === '1.00').length;
Review Comment:
These cells render through the `cellValue` passthrough (pre-computed DB values) with the default 2-decimal formatter, not `count()`, so `'1.00'` is what comes out. Tests pass as-is.
##########
superset/charts/schemas.py:
##########
@@ -1358,6 +1358,17 @@ class Meta: # pylint: disable=too-few-public-methods
load_default=False,
allow_none=True,
)
+ grouping_sets = fields.List(
+ fields.List(fields.String()),
+ metadata={
+ "description": "Rollup levels for non-additive totals: each entry is "
+ "the list of groupby columns to group at that level (e.g. the empty "
+ "list is the grand total). When set and the engine supports it, the "
+ "levels are computed in a single GROUPING SETS query.",
+ },
+ load_default=None,
+ allow_none=True,
+ )
Review Comment:
Fair concern. For this POC the levels are generated from the chart's own groupby combinations; bounding and subset-validating the raw API field is hardening I want to land with the SIP-216 work rather than here.
##########
superset-frontend/plugins/plugin-chart-pivot-table/src/react-pivottable/utilities.ts:
##########
@@ -1088,76 +1115,106 @@ class PivotData {
}
processRecord(record: PivotRecord): void {
- // this code is called in a tight loop
+ // this code is called in a tight loop.
+ // Each record is tagged (in PivotTableChart) with `rows`/`columns`: the
+ // dimension labels of the rollup level that produced it. The database has
+ // already aggregated that level, so we place the value into exactly one
+ // slot and store it verbatim (no re-aggregation). A key shorter than the
+ // full dimension list identifies a subtotal level. Records without tags
+ // (e.g. direct unit-test construction) fall back to the full dimension
+ // lists, i.e. they are treated as leaf rows.
+ const recordRows = (record.rows as unknown as string[]) ?? null;
+ const recordColumns = (record.columns as unknown as string[]) ?? null;
+ const levelRows = recordRows ?? (this.props.rows as string[]);
+ const levelColumns = recordColumns ?? (this.props.cols as string[]);
+
const colKey: string[] = [];
const rowKey: string[] = [];
- (this.props.cols as string[]).forEach((col: string) => {
+ levelColumns.forEach((col: string) => {
colKey.push(col in record ? String(record[col]) : 'null');
});
- (this.props.rows as string[]).forEach((row: string) => {
+ levelRows.forEach((row: string) => {
rowKey.push(row in record ? String(record[row]) : 'null');
});
- this.allTotal.push(record);
-
- const rowStart = this.subtotals.rowEnabled ? 1 : Math.max(1, rowKey.length);
- const colStart = this.subtotals.colEnabled ? 1 : Math.max(1, colKey.length);
-
- let isRowSubtotal;
- let isColSubtotal;
- for (let ri = rowStart; ri <= rowKey.length; ri += 1) {
- isRowSubtotal = ri < rowKey.length;
- const fRowKey = rowKey.slice(0, ri);
- const flatRowKey = flatKey(fRowKey);
- if (!this.rowTotals[flatRowKey]) {
- this.rowKeys.push(fRowKey);
- this.rowTotals[flatRowKey] = this.getFormattedAggregator(
- record,
- rowKey,
- )(this, fRowKey, []);
+ const flatRowKey = flatKey(rowKey);
+ const flatColKey = flatKey(colKey);
+
+ const isColSubtotal = colKey.length < (this.props.cols as string[]).length;
+ const isRowSubtotal = rowKey.length < (this.props.rows as string[]).length;
+
+ // Register/create the column-total slot the first time this colKey is seen.
+ if (colKey.length > 0 && !this.colTotals[flatColKey]) {
+ if (!isColSubtotal || this.subtotals.colEnabled) {
+ this.colKeys.push(colKey);
}
- this.rowTotals[flatRowKey].push(record);
- this.rowTotals[flatRowKey].isSubtotal = isRowSubtotal;
+ this.colTotals[flatColKey] = this.getFormattedAggregator(record, colKey)(
+ this,
+ [],
+ colKey,
+ );
}
-
- for (let ci = colStart; ci <= colKey.length; ci += 1) {
- isColSubtotal = ci < colKey.length;
- const fColKey = colKey.slice(0, ci);
- const flatColKey = flatKey(fColKey);
- if (!this.colTotals[flatColKey]) {
- this.colKeys.push(fColKey);
- this.colTotals[flatColKey] = this.getFormattedAggregator(
- record,
+ // Register/create the row-total slot the first time this rowKey is seen.
+ if (rowKey.length > 0 && !this.rowTotals[flatRowKey]) {
+ if (!isRowSubtotal || this.subtotals.rowEnabled) {
+ this.rowKeys.push(rowKey);
+ }
+ this.rowTotals[flatRowKey] = this.getFormattedAggregator(record, rowKey)(
+ this,
+ rowKey,
+ [],
+ );
+ }
+ // Create the body-cell slot.
+ if (rowKey.length > 0 && colKey.length > 0) {
+ if (!this.tree[flatRowKey]) {
+ this.tree[flatRowKey] = {};
+ }
+ if (!this.tree[flatRowKey][flatColKey]) {
+ this.tree[flatRowKey][flatColKey] = this.getFormattedAggregator(record)(
+ this,
+ rowKey,
colKey,
- )(this, [], fColKey);
+ );
}
+ }
+
+ // Place the value in exactly one slot, determined by the level.
+ if (rowKey.length === 0 && colKey.length === 0) {
+ this.allTotal.push(record);
+ } else if (rowKey.length === 0) {
this.colTotals[flatColKey].push(record);
this.colTotals[flatColKey].isSubtotal = isColSubtotal;
+ } else if (colKey.length === 0) {
+ this.rowTotals[flatRowKey].push(record);
+ this.rowTotals[flatRowKey].isSubtotal = isRowSubtotal;
+ } else {
+ this.tree[flatRowKey][flatColKey].push(record);
+ this.tree[flatRowKey][flatColKey].isRowSubtotal = isRowSubtotal;
+ this.tree[flatRowKey][flatColKey].isColSubtotal = isColSubtotal;
+ this.tree[flatRowKey][flatColKey].isSubtotal =
+ isRowSubtotal || isColSubtotal;
}
- // And now fill in for all the sub-cells.
- for (let ri = rowStart; ri <= rowKey.length; ri += 1) {
- isRowSubtotal = ri < rowKey.length;
- const fRowKey = rowKey.slice(0, ri);
- const flatRowKey = flatKey(fRowKey);
- if (!this.tree[flatRowKey]) {
- this.tree[flatRowKey] = {};
+ // Metric-collapse totals. The metric is a pseudo-dimension always present on
+ // one axis, so no rollup level produces an empty key on that axis -- which
+ // would leave the opposite "Total" axis and the grand-total corner empty.
+ // When a record's axis holds only the metric (no real dims there), its value
+ // is also the collapsed total for that axis, so mirror it into rowTotals /
+ // colTotals / allTotal. (For a single metric this equals the metric column;
+ // for multiple metrics it is the last metric -- a cross-metric total is not
+ // well defined and is left as future work.)
+ const metricKey = record.__metricKey as unknown as string | undefined;
+ if (metricKey) {
+ const realColCount = levelColumns.filter(c => c !== metricKey).length;
+ const realRowCount = levelRows.filter(r => r !== metricKey).length;
+ if (levelColumns.includes(metricKey) && realColCount === 0) {
+ if (rowKey.length === 0) this.allTotal.push(record);
+ else this.rowTotals[flatRowKey]?.push(record);
}
- for (let ci = colStart; ci <= colKey.length; ci += 1) {
- isColSubtotal = ci < colKey.length;
- const fColKey = colKey.slice(0, ci);
- const flatColKey = flatKey(fColKey);
- if (!this.tree[flatRowKey][flatColKey]) {
- this.tree[flatRowKey][flatColKey] = this.getFormattedAggregator(
- record,
- )(this, fRowKey, fColKey);
- }
- this.tree[flatRowKey][flatColKey].push(record);
-
- this.tree[flatRowKey][flatColKey].isRowSubtotal = isRowSubtotal;
- this.tree[flatRowKey][flatColKey].isColSubtotal = isColSubtotal;
- this.tree[flatRowKey][flatColKey].isSubtotal =
- isRowSubtotal || isColSubtotal;
+ if (levelRows.includes(metricKey) && realRowCount === 0) {
+ if (colKey.length === 0) this.allTotal.push(record);
+ else this.colTotals[flatColKey]?.push(record);
}
Review Comment:
Right, and the comment right above calls it out: a cross-metric total isn't well-defined, so it's intentionally the last metric for the POC and left for the SIP.
##########
superset-frontend/plugins/plugin-chart-pivot-table/src/PivotTableChart.tsx:
##########
@@ -341,22 +296,43 @@ export default function PivotTableChart(props: PivotTableProps) {
const unpivotedData = useMemo(
() =>
- data.reduce(
- (acc: Record<string, any>[], record: Record<string, any>) => [
- ...acc,
- ...metricNames
+ // `data` is now one entry per rollup level. Tag every record with the
+ // row/column dimension labels of the level that produced it (mirroring
+ // the METRIC_KEY injection used for the full rows/cols below) so
+ // PivotData can slot each pre-computed value without re-aggregating.
+ // buildGroupbyCombinations already applied transposePivot, so the level's
+ // groupby is display-oriented and is not transposed again here.
+ data.flatMap((query: QueryData) => {
+ let levelRows = query.groupby.rows.map(getColumnLabel);
+ let levelCols = query.groupby.columns.map(getColumnLabel);
+ if (metricsLayout === MetricsLayoutEnum.ROWS) {
+ levelRows = combineMetric
+ ? [...levelRows, METRIC_KEY]
+ : [METRIC_KEY, ...levelRows];
+ } else {
+ levelCols = combineMetric
+ ? [...levelCols, METRIC_KEY]
+ : [METRIC_KEY, ...levelCols];
+ }
+ return query.data.flatMap((record: Record<string, any>) =>
+ metricNames
.map((name: string) => ({
...record,
[METRIC_KEY]: name,
value: record[name],
// Mark currency column for per-cell currency detection in aggregators
__currencyColumn: currencyCodeColumn,
+ // The level this record belongs to (used by PivotData placement).
+ rows: levelRows,
+ columns: levelCols,
+ // Identify the metric pseudo-dimension so PivotData can feed the
+ // metric-collapsed totals (the opposite "Total" axis + corner).
+ __metricKey: METRIC_KEY,
}))
- .filter(record => record.value !== null),
- ],
- [],
- ),
- [data, metricNames, currencyCodeColumn],
+ .filter(r => r.value !== null),
Review Comment:
The filter drops the null-metric rows the rollup/GROUPING SETS split emits per level. A metric that's legitimately null at a level getting pruned is a real edge, but one I'm leaving for the SIP work.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(table/pivot-table): correct non-additive totals/subtotals via DB rollup [POC for SIP-216] [superset]
Posted by "bito-code-review[bot] (via GitHub)" <gi...@apache.org>.
bito-code-review[bot] commented on code in PR #41184:
URL: https://github.com/apache/superset/pull/41184#discussion_r3565603918
##########
superset-frontend/plugins/plugin-chart-pivot-table/src/plugin/buildQuery.ts:
##########
@@ -54,6 +59,39 @@ export default function buildQuery(formData: PivotTableQueryFormData) {
}
return col;
});
+}
+
+export default function buildQuery(formData: PivotTableQueryFormData) {
+ const { extra_form_data } = formData;
+ const time_grain_sqla =
+ extra_form_data?.time_grain_sqla || formData.time_grain_sqla;
Review Comment:
<div>
<div id="suggestion">
<div id="issue"><b>Dashboard time grain override not propagated</b></div>
<div id="fix">
The `time_grain_sqla` value resolved from `extra_form_data` (lines 66–67) is used to build `getQueryColumns` columns but is never forwarded to `extras.time_grain_sqla` on the query object returned at line 109. Dashboard-level time grain overrides are therefore not applied by the database, despite being extracted. The existing test at line 178 (`expect(query.extras?.time_grain_sqla).toEqual(...)`) will catch this regression once the fix is in place.
</div>
</div>
<small><i>Code Review Run #90e9f1</i></small>
</div>
---
Should Bito avoid suggestions like this for future reviews? (<a href=https://alpha.bito.ai/home/ai-agents/review-rules>Manage Rules</a>)
- [ ] Yes, avoid them
##########
superset-frontend/plugins/plugin-chart-pivot-table/test/react-pivottable/tableRenders.test.tsx:
##########
@@ -149,18 +202,83 @@ test('TableRenderer renders col totals row when colTotals is enabled', () => {
test('TableRenderer renders grand total when both totals are enabled', () => {
const props = buildDefaultProps({
+ data: TAGGED_COUNT_DATA,
+ vals: ['value'],
tableOptions: { rowTotals: true, colTotals: true },
});
renderWithTheme(<TableRenderer {...props} />);
- // The grand total cell should show "4" (total record count).
+ // The grand total cell shows the DB-computed "4" (total record count).
const grandTotalCells = screen
.getAllByRole('gridcell')
.filter(cell => cell.classList.contains('pvtGrandTotal'));
expect(grandTotalCells.length).toBe(1);
expect(grandTotalCells[0]).toHaveTextContent('4');
Review Comment:
<div>
<div id="suggestion">
<div id="issue"><b>Wrong assertion format</b></div>
<div id="fix">
The grand total assertion expects raw `'4'` but the formatter (default: `numberFormat` with `digitsAfterDecimal: 2`) will render `4` as `'4.00'`. Other tests in this diff correctly use `'1.00'` and `'2.00'`. This assertion will fail at runtime.
</div>
</div>
<small><i>Code Review Run #90e9f1</i></small>
</div>
---
Should Bito avoid suggestions like this for future reviews? (<a href=https://alpha.bito.ai/home/ai-agents/review-rules>Manage Rules</a>)
- [ ] Yes, avoid them
##########
superset-frontend/plugins/plugin-chart-pivot-table/src/react-pivottable/TableRenderers.tsx:
##########
@@ -94,7 +94,6 @@ interface SubtotalOptions {
interface TableRendererProps {
cols: string[];
rows: string[];
- aggregatorName: string;
tableOptions?: TableOptions;
Review Comment:
<div>
<div id="suggestion">
<div id="issue"><b>CWE-693: Protection Mechanism Bypassed</b></div>
<div id="fix">
Add `aggregatorName: string;` back to the TableRendererProps interface and reintroduce `aggregatorName` in props destructuring so it can be used in total headers. (See also: [CWE-693](https://cwe.mitre.org/data/definitions/693.html))
</div>
</div>
<small><i>Code Review Run #90e9f1</i></small>
</div>
---
Should Bito avoid suggestions like this for future reviews? (<a href=https://alpha.bito.ai/home/ai-agents/review-rules>Manage Rules</a>)
- [ ] Yes, avoid them
##########
superset-frontend/plugins/plugin-chart-pivot-table/src/react-pivottable/TableRenderers.tsx:
##########
@@ -1099,9 +1096,7 @@ export function TableRenderer(props: TableRendererProps) {
true,
)}
>
- {t('Total (%(aggregatorName)s)', {
- aggregatorName: t(aggregatorName),
- })}
+ {t('Total')}
Review Comment:
<div>
<div id="suggestion">
<div id="issue"><b>CWE-693: Protection Mechanism Bypassed</b></div>
<div id="fix">
Restore the translation pattern `{t('Total (%(aggregatorName)s)', { aggregatorName: t(aggregatorName) })}` for the column grand total header after ensuring `aggregatorName` is available in props. (See also: [CWE-693](https://cwe.mitre.org/data/definitions/693.html))
</div>
</div>
<small><i>Code Review Run #90e9f1</i></small>
</div>
---
Should Bito avoid suggestions like this for future reviews? (<a href=https://alpha.bito.ai/home/ai-agents/review-rules>Manage Rules</a>)
- [ ] Yes, avoid them
##########
superset-frontend/plugins/plugin-chart-pivot-table/src/react-pivottable/TableRenderers.tsx:
##########
@@ -1183,11 +1177,7 @@ export function TableRenderer(props: TableRendererProps) {
true,
)}
>
- {settingsColAttrs.length === 0
- ? t('Total (%(aggregatorName)s)', {
- aggregatorName: t(aggregatorName),
- })
- : null}
+ {settingsColAttrs.length === 0 ? t('Total') : null}
Review Comment:
<div>
<div id="suggestion">
<div id="issue"><b>CWE-693: Protection Mechanism Bypassed</b></div>
<div id="fix">
Restore the translation pattern `{settingsColAttrs.length === 0 ? t('Total (%(aggregatorName)s)', { aggregatorName: t(aggregatorName) }) : null}` for the row grand total header, after making `aggregatorName` available. (See also: [CWE-693](https://cwe.mitre.org/data/definitions/693.html))
</div>
</div>
<small><i>Code Review Run #90e9f1</i></small>
</div>
---
Should Bito avoid suggestions like this for future reviews? (<a href=https://alpha.bito.ai/home/ai-agents/review-rules>Manage Rules</a>)
- [ ] Yes, avoid them
##########
superset-frontend/plugins/plugin-chart-pivot-table/src/react-pivottable/TableRenderers.tsx:
##########
@@ -1448,9 +1437,7 @@ export function TableRenderer(props: TableRendererProps) {
true,
)}
>
- {t('Total (%(aggregatorName)s)', {
- aggregatorName: t(aggregatorName),
- })}
+ {t('Total')}
Review Comment:
<div>
<div id="suggestion">
<div id="issue"><b>CWE-693: Protection Mechanism Bypassed</b></div>
<div id="fix">
Restore `{t('Total (%(aggregatorName)s)', { aggregatorName: t(aggregatorName) })}` for the row total column header after reintroducing `aggregatorName` in props. (See also: [CWE-693](https://cwe.mitre.org/data/definitions/693.html))
</div>
</div>
<small><i>Code Review Run #90e9f1</i></small>
</div>
---
Should Bito avoid suggestions like this for future reviews? (<a href=https://alpha.bito.ai/home/ai-agents/review-rules>Manage Rules</a>)
- [ ] Yes, avoid them
##########
superset-frontend/plugins/plugin-chart-pivot-table/src/react-pivottable/utilities.ts:
##########
@@ -1088,76 +1117,108 @@ class PivotData {
}
processRecord(record: PivotRecord): void {
- // this code is called in a tight loop
+ // this code is called in a tight loop.
+ // Each record is tagged (in PivotTableChart) with `__rows`/`__columns`:
+ // the dimension labels of the rollup level that produced it. The database
+ // has already aggregated that level, so we place the value into exactly
+ // one slot and store it verbatim (no re-aggregation). A key shorter than
+ // the full dimension list identifies a subtotal level. Records without
+ // tags (e.g. direct unit-test construction) fall back to the full
+ // dimension lists, i.e. they are treated as leaf rows. The `__` prefix
+ // keeps these rollup tags from colliding with a real dataset column
+ // named `rows`/`columns`.
+ const recordRows = (record.__rows as unknown as string[]) ?? null;
+ const recordColumns = (record.__columns as unknown as string[]) ?? null;
+ const levelRows = recordRows ?? (this.props.rows as string[]);
+ const levelColumns = recordColumns ?? (this.props.cols as string[]);
+
const colKey: string[] = [];
const rowKey: string[] = [];
- (this.props.cols as string[]).forEach((col: string) => {
+ levelColumns.forEach((col: string) => {
colKey.push(col in record ? String(record[col]) : 'null');
});
- (this.props.rows as string[]).forEach((row: string) => {
+ levelRows.forEach((row: string) => {
rowKey.push(row in record ? String(record[row]) : 'null');
});
- this.allTotal.push(record);
-
- const rowStart = this.subtotals.rowEnabled ? 1 : Math.max(1, rowKey.length);
- const colStart = this.subtotals.colEnabled ? 1 : Math.max(1, colKey.length);
-
- let isRowSubtotal;
- let isColSubtotal;
- for (let ri = rowStart; ri <= rowKey.length; ri += 1) {
- isRowSubtotal = ri < rowKey.length;
- const fRowKey = rowKey.slice(0, ri);
- const flatRowKey = flatKey(fRowKey);
- if (!this.rowTotals[flatRowKey]) {
- this.rowKeys.push(fRowKey);
- this.rowTotals[flatRowKey] = this.getFormattedAggregator(
- record,
- rowKey,
- )(this, fRowKey, []);
+ const flatRowKey = flatKey(rowKey);
+ const flatColKey = flatKey(colKey);
+
+ const isColSubtotal = colKey.length < (this.props.cols as string[]).length;
+ const isRowSubtotal = rowKey.length < (this.props.rows as string[]).length;
+
+ // Register/create the column-total slot the first time this colKey is seen.
+ if (colKey.length > 0 && !this.colTotals[flatColKey]) {
+ if (!isColSubtotal || this.subtotals.colEnabled) {
+ this.colKeys.push(colKey);
}
- this.rowTotals[flatRowKey].push(record);
- this.rowTotals[flatRowKey].isSubtotal = isRowSubtotal;
+ this.colTotals[flatColKey] = this.getFormattedAggregator(record, colKey)(
+ this,
+ [],
+ colKey,
+ );
}
-
- for (let ci = colStart; ci <= colKey.length; ci += 1) {
- isColSubtotal = ci < colKey.length;
- const fColKey = colKey.slice(0, ci);
- const flatColKey = flatKey(fColKey);
- if (!this.colTotals[flatColKey]) {
- this.colKeys.push(fColKey);
- this.colTotals[flatColKey] = this.getFormattedAggregator(
- record,
+ // Register/create the row-total slot the first time this rowKey is seen.
+ if (rowKey.length > 0 && !this.rowTotals[flatRowKey]) {
+ if (!isRowSubtotal || this.subtotals.rowEnabled) {
+ this.rowKeys.push(rowKey);
+ }
+ this.rowTotals[flatRowKey] = this.getFormattedAggregator(record, rowKey)(
+ this,
+ rowKey,
+ [],
+ );
+ }
+ // Create the body-cell slot.
+ if (rowKey.length > 0 && colKey.length > 0) {
+ if (!this.tree[flatRowKey]) {
+ this.tree[flatRowKey] = {};
+ }
+ if (!this.tree[flatRowKey][flatColKey]) {
+ this.tree[flatRowKey][flatColKey] = this.getFormattedAggregator(record)(
+ this,
+ rowKey,
colKey,
- )(this, [], fColKey);
+ );
}
+ }
+
+ // Place the value in exactly one slot, determined by the level.
+ if (rowKey.length === 0 && colKey.length === 0) {
+ this.allTotal.push(record);
+ } else if (rowKey.length === 0) {
this.colTotals[flatColKey].push(record);
this.colTotals[flatColKey].isSubtotal = isColSubtotal;
+ } else if (colKey.length === 0) {
+ this.rowTotals[flatRowKey].push(record);
+ this.rowTotals[flatRowKey].isSubtotal = isRowSubtotal;
+ } else {
+ this.tree[flatRowKey][flatColKey].push(record);
+ this.tree[flatRowKey][flatColKey].isRowSubtotal = isRowSubtotal;
+ this.tree[flatRowKey][flatColKey].isColSubtotal = isColSubtotal;
+ this.tree[flatRowKey][flatColKey].isSubtotal =
+ isRowSubtotal || isColSubtotal;
}
- // And now fill in for all the sub-cells.
- for (let ri = rowStart; ri <= rowKey.length; ri += 1) {
- isRowSubtotal = ri < rowKey.length;
- const fRowKey = rowKey.slice(0, ri);
- const flatRowKey = flatKey(fRowKey);
- if (!this.tree[flatRowKey]) {
- this.tree[flatRowKey] = {};
+ // Metric-collapse totals. The metric is a pseudo-dimension always present on
+ // one axis, so no rollup level produces an empty key on that axis -- which
+ // would leave the opposite "Total" axis and the grand-total corner empty.
+ // When a record's axis holds only the metric (no real dims there), its value
+ // is also the collapsed total for that axis, so mirror it into rowTotals /
+ // colTotals / allTotal. (For a single metric this equals the metric column;
+ // for multiple metrics it is the last metric -- a cross-metric total is not
+ // well defined and is left as future work.)
+ const metricKey = record.__metricKey as unknown as string | undefined;
+ if (metricKey) {
+ const realColCount = levelColumns.filter(c => c !== metricKey).length;
+ const realRowCount = levelRows.filter(r => r !== metricKey).length;
+ if (levelColumns.includes(metricKey) && realColCount === 0) {
+ if (rowKey.length === 0) this.allTotal.push(record);
+ else this.rowTotals[flatRowKey]?.push(record);
}
- for (let ci = colStart; ci <= colKey.length; ci += 1) {
- isColSubtotal = ci < colKey.length;
- const fColKey = colKey.slice(0, ci);
- const flatColKey = flatKey(fColKey);
- if (!this.tree[flatRowKey][flatColKey]) {
- this.tree[flatRowKey][flatColKey] = this.getFormattedAggregator(
- record,
- )(this, fRowKey, fColKey);
- }
- this.tree[flatRowKey][flatColKey].push(record);
-
- this.tree[flatRowKey][flatColKey].isRowSubtotal = isRowSubtotal;
- this.tree[flatRowKey][flatColKey].isColSubtotal = isColSubtotal;
- this.tree[flatRowKey][flatColKey].isSubtotal =
- isRowSubtotal || isColSubtotal;
+ if (levelRows.includes(metricKey) && realRowCount === 0) {
+ if (colKey.length === 0) this.allTotal.push(record);
+ else this.colTotals[flatColKey]?.push(record);
}
Review Comment:
<div>
<div id="suggestion">
<div id="issue"><b>Unguarded metric-collapse totals</b></div>
<div id="fix">
Metric-collapse logic (lines 1211-1222) runs unconditionally even when subtotals are disabled, contradicting the subtotal-guard pattern established at lines 1151-1160. Add this.subtotals row/colEnabled checks.
</div>
</div>
<small><i>Code Review Run #90e9f1</i></small>
</div>
---
Should Bito avoid suggestions like this for future reviews? (<a href=https://alpha.bito.ai/home/ai-agents/review-rules>Manage Rules</a>)
- [ ] Yes, avoid them
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(table/pivot-table): correct non-additive totals/subtotals via DB rollup [POC for SIP-216] [superset]
Posted by "bito-code-review[bot] (via GitHub)" <gi...@apache.org>.
bito-code-review[bot] commented on PR #41184:
URL: https://github.com/apache/superset/pull/41184#issuecomment-4952447448
AI Code Review is in progress (usually takes 3 to 15 minutes unless it's a very large PR).
<details>
<summary>
Bito Usage Guide
</summary>
**Commands**
Type the following command in the pull request comment and save the comment.
- `/review` - Manually triggers a full AI review.
- `/pause` - Pauses automatic reviews on this pull request.
- `/resume` - Resumes automatic reviews.
- `/resolve` - Marks all Bito-posted review comments as resolved.
- `/abort` - Cancels all in-progress reviews.
Refer to the <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/available-commands" target="_blank" rel="noopener noreferrer">documentation</a> for additional commands.
**Configuration**
This repository uses `Superset` You can customize the agent settings <a href="https://alpha.bito.ai/home/ai-agents/code-review-agent" target="_blank" rel="noopener noreferrer">here</a> or contact your Bito workspace admin at evan@preset.io.
**Documentation & Help**
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/getting-started/install-run-using-bito-cloud/create-or-customize-an-agent-instance" target="_blank" rel="noopener noreferrer">Customize agent settings</a>
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/implementing-custom-code-review-rules" target="_blank" rel="noopener noreferrer">Review rules</a>
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent" target="_blank" rel="noopener noreferrer">General documentation</a>
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/faqs" target="_blank" rel="noopener noreferrer">FAQ</a>
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] [SIP] Correct totals/subtotals for non-additive metrics (Table + Pivot) [superset]
Posted by "bito-code-review[bot] (via GitHub)" <gi...@apache.org>.
bito-code-review[bot] commented on PR #41184:
URL: https://github.com/apache/superset/pull/41184#issuecomment-4738578841
AI Code Review is in progress (usually takes 3 to 15 minutes unless it's a very large PR).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(table/pivot-table): correct non-additive totals/subtotals via DB rollup [POC for SIP-216] [superset]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #41184:
URL: https://github.com/apache/superset/pull/41184#issuecomment-4988712622
🎪 [Showtime](https://github.com/mistercrunch/superset-showtime) deployed environment on [GHA](https://github.com/apache/superset/actions/runs/29474785437) for [67b8a00](https://github.com/apache/superset/commit/67b8a00)
• **Environment:** http://44.246.215.91:8080 (admin/admin)
• **Lifetime:** 48h auto-cleanup
• **Updates:** New commits create fresh environments automatically
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(table/pivot-table): correct non-additive totals/subtotals via DB rollup [POC for SIP-216] [superset]
Posted by "codeant-ai-for-open-source[bot] (via GitHub)" <gi...@apache.org>.
codeant-ai-for-open-source[bot] commented on code in PR #41184:
URL: https://github.com/apache/superset/pull/41184#discussion_r3562183687
##########
superset-frontend/plugins/plugin-chart-pivot-table/src/PivotTableChart.tsx:
##########
@@ -341,22 +296,43 @@ export default function PivotTableChart(props: PivotTableProps) {
const unpivotedData = useMemo(
() =>
- data.reduce(
- (acc: Record<string, any>[], record: Record<string, any>) => [
- ...acc,
- ...metricNames
+ // `data` is now one entry per rollup level. Tag every record with the
+ // row/column dimension labels of the level that produced it (mirroring
+ // the METRIC_KEY injection used for the full rows/cols below) so
+ // PivotData can slot each pre-computed value without re-aggregating.
+ // buildGroupbyCombinations already applied transposePivot, so the level's
+ // groupby is display-oriented and is not transposed again here.
+ data.flatMap((query: QueryData) => {
+ let levelRows = query.groupby.rows.map(getColumnLabel);
+ let levelCols = query.groupby.columns.map(getColumnLabel);
+ if (metricsLayout === MetricsLayoutEnum.ROWS) {
+ levelRows = combineMetric
+ ? [...levelRows, METRIC_KEY]
+ : [METRIC_KEY, ...levelRows];
+ } else {
+ levelCols = combineMetric
+ ? [...levelCols, METRIC_KEY]
+ : [METRIC_KEY, ...levelCols];
+ }
+ return query.data.flatMap((record: Record<string, any>) =>
+ metricNames
.map((name: string) => ({
...record,
[METRIC_KEY]: name,
value: record[name],
// Mark currency column for per-cell currency detection in aggregators
__currencyColumn: currencyCodeColumn,
+ // The level this record belongs to (used by PivotData placement).
+ rows: levelRows,
+ columns: levelCols,
Review Comment:
**Suggestion:** Using plain `rows` and `columns` as internal metadata keys can overwrite real dataset fields with the same names, which corrupts grouping keys and cross-filter payloads for charts whose source columns are named `rows` or `columns`. Store these rollup tags under reserved internal keys (for example `__rows` / `__columns`) and read those keys in PivotData instead. [incorrect variable usage]
<details>
<summary><b>Severity Level:</b> Major ⚠️</summary>
```mdx
- ❌ Pivot tables misplace cells for `rows`/`columns` dimensions.
- ⚠️ Cross-filter callbacks send wrong filter values to backend.
- ⚠️ Totals/subtotals inconsistent when dimension names collide.
```
</details>
<details>
<summary><b>Steps of Reproduction ✅ </b></summary>
```mdx
1. In Superset, create a Pivot Table chart using a dataset that has a physical column
named `rows` or `columns`, and include that column in the chart’s groupby (so it appears
in `formData.groupbyRows` or `formData.groupbyColumns`). This chart type is wired to the
`PivotTableChart` visualization via the plugin entrypoint at
`superset-frontend/plugins/plugin-chart-pivot-table/src/plugin/index.ts:36-75`, which
calls `loadChart: () => import('../PivotTableChart')`.
2. When the chart queries run, the backend response is converted into `DataRecord` rows
where each column alias becomes a key, including any column named `rows` or `columns`.
This is documented in `transformProps` at
`superset-frontend/plugins/plugin-chart-pivot-table/src/plugin/transformProps.ts:73-76`,
and those `DataRecord[]` arrays are wrapped into `QueryData` objects (`types.ts:65-68`)
and passed to `PivotTableChart` as the `data: QueryData[]` prop
(`transformProps.ts:216-220`).
3. Inside `PivotTableChart`, the `unpivotedData` memo at
`superset-frontend/plugins/plugin-chart-pivot-table/src/PivotTableChart.tsx:297-335`
flattens each `QueryData` level. For every underlying `record: DataRecord`, it builds a
new object with `...record` and then assigns rollup metadata: `rows: levelRows` and
`columns: levelCols` at lines `325-327`. If the original `record` already had a field
named `rows` or `columns` (because the dataset column was named that), these assignments
at `326-327` overwrite the real data values with the rollup tag arrays.
4. During rendering, `TableRenderer` creates a `PivotData` instance
(`TableRenderers.tsx:104-110`), which iterates records in
`react-pivottable/utilities.ts:1120-1239`. There, `record.rows` and `record.columns` are
read as rollup tags (`const recordRows = (record.rows as string[]) ?? null; const
recordColumns = (record.columns as string[]) ?? null;` at lines `1126-1129`), and
`levelRows` / `levelColumns` are derived from them. For a dimension whose label is
literally `'rows'` or `'columns'`, the aggregator then builds row/column keys using
`record[row]` / `record[col]` (`utilities.ts:1134-1139`), but because `record.rows` /
`record.columns` were overwritten with the tag arrays in step 3, the key values for those
dimensions become metadata strings instead of the true cell values. These corrupted keys
propagate into click handlers in `TableRenderers.tsx:94-120` and header click handlers in
`TableRenderers.tsx:124-152`, which construct `filters: Record<string,string>` from those
row/column values and pass them to `PivotTableChart.toggleFilter`
(`PivotTableChart.tsx:493-551`). As a result, for charts whose source columns are named
`rows` or `columns`, pivot placement and emitted cross-filter payloads are incorrect
because the metadata tags have silently overwritten the real data.
```
</details>
[](https://app.codeant.ai/fix-in-ide?tool=cursor&prompt_id=d5d63c6edb524ddf9352c6a738edecf8&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset) [](https://app.codeant.ai/fix-in-ide?tool=vscode-claude&prompt_id=d5d63c6edb524ddf9352c6a738edecf8&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset)
*(Use Cmd/Ctrl + Click for best experience)*
<details>
<summary><b>Prompt for AI Agent 🤖 </b></summary>
```mdx
This is a comment left during a code review.
**Path:** superset-frontend/plugins/plugin-chart-pivot-table/src/PivotTableChart.tsx
**Line:** 326:327
**Comment:**
*Incorrect Variable Usage: Using plain `rows` and `columns` as internal metadata keys can overwrite real dataset fields with the same names, which corrupts grouping keys and cross-filter payloads for charts whose source columns are named `rows` or `columns`. Store these rollup tags under reserved internal keys (for example `__rows` / `__columns`) and read those keys in PivotData instead.
Validate the correctness of the flagged issue. If correct, How can I resolve this? If you propose a fix, implement it and please make it concise.
Once fix is implemented, also check other comments on the same PR, and ask user if the user wants to fix the rest of the comments as well. if said yes, then fetch all the comments validate the correctness and implement a minimal fix
```
</details>
<a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F41184&comment_hash=903c7ef006ee394b728e882a0ef42a04153c3989806ab41decf2d37ccd03b365&reaction=like'>👍</a> | <a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F41184&comment_hash=903c7ef006ee394b728e882a0ef42a04153c3989806ab41decf2d37ccd03b365&reaction=dislike'>👎</a>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(table/pivot-table): correct non-additive totals/subtotals via DB rollup [POC for SIP-216] [superset]
Posted by "rusackas (via GitHub)" <gi...@apache.org>.
rusackas commented on code in PR #41184:
URL: https://github.com/apache/superset/pull/41184#discussion_r3562158513
##########
superset/common/query_context_processor.py:
##########
@@ -252,9 +254,54 @@ def get_query_result(self, query_object: QueryObject) -> QueryResult:
This method delegates to the datasource's get_query_result method,
which handles query execution, normalization, time offsets, and
post-processing.
+
+ When the query requests rollup ``grouping_sets`` but the engine does not
+ support native ``GROUPING SETS``, fall back to one query per level and
+ concatenate the results with ``GROUPING()``-equivalent markers, so the
+ combined result matches the shape the native path produces (SIP.md,
+ phase 3b). Engines that support it run the single native query.
"""
+ if query_object.grouping_sets and not self._supports_grouping_sets():
+ return self._grouping_sets_fallback(query_object)
return self._qc_datasource.get_query_result(query_object)
+ def _supports_grouping_sets(self) -> bool:
+ engine_spec = getattr(self._qc_datasource, "db_engine_spec", None)
+ return bool(engine_spec and engine_spec.supports_grouping_sets)
+
+ def _grouping_sets_fallback(self, query_object: QueryObject) -> QueryResult:
+ """
+ Emulate a GROUPING SETS query on engines without native support: run one
+ query per rollup level and concatenate, tagging each level's rows with
+ the same per-column markers the native path emits.
+ """
+ levels = query_object.grouping_sets
+ all_labels = get_column_names_from_columns(query_object.columns)
+ label_to_column = dict(zip(all_labels, query_object.columns, strict=False))
Review Comment:
Good catch, fixed. `all_labels` now uses `get_column_name` per column so it lines up positionally with `query_object.columns`, and the zip is strict so a mismatch would blow up instead of silently mapping the wrong column.
##########
superset/common/query_context_processor.py:
##########
@@ -252,9 +254,54 @@ def get_query_result(self, query_object: QueryObject) -> QueryResult:
This method delegates to the datasource's get_query_result method,
which handles query execution, normalization, time offsets, and
post-processing.
+
+ When the query requests rollup ``grouping_sets`` but the engine does not
+ support native ``GROUPING SETS``, fall back to one query per level and
+ concatenate the results with ``GROUPING()``-equivalent markers, so the
+ combined result matches the shape the native path produces (SIP.md,
+ phase 3b). Engines that support it run the single native query.
"""
+ if query_object.grouping_sets and not self._supports_grouping_sets():
+ return self._grouping_sets_fallback(query_object)
return self._qc_datasource.get_query_result(query_object)
+ def _supports_grouping_sets(self) -> bool:
+ engine_spec = getattr(self._qc_datasource, "db_engine_spec", None)
+ return bool(engine_spec and engine_spec.supports_grouping_sets)
+
+ def _grouping_sets_fallback(self, query_object: QueryObject) -> QueryResult:
+ """
+ Emulate a GROUPING SETS query on engines without native support: run one
+ query per rollup level and concatenate, tagging each level's rows with
+ the same per-column markers the native path emits.
+ """
+ levels = query_object.grouping_sets
+ all_labels = get_column_names_from_columns(query_object.columns)
+ label_to_column = dict(zip(all_labels, query_object.columns, strict=False))
+
+ frames: list[pd.DataFrame] = []
+ result: QueryResult | None = None
+ for level in levels:
+ level_labels = set(level)
+ sub_query = copy.copy(query_object)
+ sub_query.grouping_sets = []
+ sub_query.columns = [
+ label_to_column[label] for label in all_labels if label in level_labels
+ ]
Review Comment:
Good catch, fixed. `all_labels` is derived the same way the native path labels adhoc columns, so those groupby columns actually make it into the fallback subqueries instead of getting silently dropped.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(table/pivot-table): correct non-additive totals/subtotals via DB rollup [POC for SIP-216] [superset]
Posted by "codeant-ai-for-open-source[bot] (via GitHub)" <gi...@apache.org>.
codeant-ai-for-open-source[bot] commented on code in PR #41184:
URL: https://github.com/apache/superset/pull/41184#discussion_r3557779929
##########
superset/common/query_context_processor.py:
##########
@@ -252,9 +254,54 @@ def get_query_result(self, query_object: QueryObject) -> QueryResult:
This method delegates to the datasource's get_query_result method,
which handles query execution, normalization, time offsets, and
post-processing.
+
+ When the query requests rollup ``grouping_sets`` but the engine does not
+ support native ``GROUPING SETS``, fall back to one query per level and
+ concatenate the results with ``GROUPING()``-equivalent markers, so the
+ combined result matches the shape the native path produces (SIP.md,
+ phase 3b). Engines that support it run the single native query.
"""
+ if query_object.grouping_sets and not self._supports_grouping_sets():
+ return self._grouping_sets_fallback(query_object)
return self._qc_datasource.get_query_result(query_object)
+ def _supports_grouping_sets(self) -> bool:
+ engine_spec = getattr(self._qc_datasource, "db_engine_spec", None)
+ return bool(engine_spec and engine_spec.supports_grouping_sets)
+
+ def _grouping_sets_fallback(self, query_object: QueryObject) -> QueryResult:
+ """
+ Emulate a GROUPING SETS query on engines without native support: run one
+ query per rollup level and concatenate, tagging each level's rows with
+ the same per-column markers the native path emits.
+ """
+ levels = query_object.grouping_sets
+ all_labels = get_column_names_from_columns(query_object.columns)
+ label_to_column = dict(zip(all_labels, query_object.columns, strict=False))
Review Comment:
**Suggestion:** The fallback builds `label_to_column` by zipping a filtered label list with the unfiltered `query_object.columns` list, which misaligns entries whenever any non-physical/adhoc column is present. That can map a label to the wrong column object and generate incorrect grouping SQL. Build the mapping from the original columns and their actual labels in the same pass (without dropping entries first) so each label always points to its correct column definition. [incorrect variable usage]
<details>
<summary><b>Severity Level:</b> Critical 🚨</summary>
```mdx
- ❌ Grouping_sets fallback groups by wrong columns on SQLite.
- ⚠️ Pivot/Table non-additive totals incorrect on fallback engines.
```
</details>
<details>
<summary><b>Steps of Reproduction ✅ </b></summary>
```mdx
1. Build a Chart Data payload similar to
`TestGroupingSetsRollup.test_grouping_sets_returns_all_levels` in
`tests/integration_tests/non_additive_totals_tests.py:12-32`, but set
`queries[0]["columns"]` to include an adhoc SQL column dict first and a physical column
second (e.g. `columns = [{"expressionType": "SQL", "sqlExpression": "...", "label":
"adhoc_col"}, "state"]`), and define `grouping_sets` levels that reference both labels
(e.g. `[["adhoc_col", "state"], ["adhoc_col"], []]`).
2. Send this payload to `ChartDataQueryContextSchema().load(...)` (see
`superset/charts/schemas.py:32-42` and
`tests/integration_tests/non_additive_totals_tests.py:81-84`), which constructs a
`QueryContext` and its first `QueryObject` via `QueryObjectFactory.create`
(`superset/common/query_object_factory.py:52-95`). The resulting `QueryObject.columns`
list preserves the adhoc + physical ordering (`superset/common/query_object.py:114-162`).
3. Run the query against a datasource whose `db_engine_spec.supports_grouping_sets` is
false (e.g. SQLite, referenced in the docstring at
`tests/integration_tests/non_additive_totals_tests.py:4-10`).
`QueryContext.get_query_result()` (`superset/common/query_context.py:134-135`) delegates
to `QueryContextProcessor.get_query_result()`
(`superset/common/query_context_processor.py:250-266`), which detects
`query_object.grouping_sets` and calls `_grouping_sets_fallback()` when native grouping
sets are unsupported.
4. In `_grouping_sets_fallback()` (`superset/common/query_context_processor.py:272-281`),
`get_column_names_from_columns(query_object.columns)` from
`superset/utils/core.py:1699-1707` drops adhoc columns, producing a shorter `all_labels`
list. `dict(zip(all_labels, query_object.columns, strict=False))` then pairs each physical
label with the first N entries of the full `columns` list, so when an adhoc appears before
a later physical column the label maps to the wrong `Column` object. Subsequent per-level
subqueries built from `label_to_column` group by mis-mapped columns, yielding rollup
results and grouping markers that no longer correspond to the intended dimensions.
```
</details>
[](https://app.codeant.ai/fix-in-ide?tool=cursor&prompt_id=b5b0140dab1c4ec2a832d9b48829d409&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset) [](https://app.codeant.ai/fix-in-ide?tool=vscode-claude&prompt_id=b5b0140dab1c4ec2a832d9b48829d409&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset)
*(Use Cmd/Ctrl + Click for best experience)*
<details>
<summary><b>Prompt for AI Agent 🤖 </b></summary>
```mdx
This is a comment left during a code review.
**Path:** superset/common/query_context_processor.py
**Line:** 279:280
**Comment:**
*Incorrect Variable Usage: The fallback builds `label_to_column` by zipping a filtered label list with the unfiltered `query_object.columns` list, which misaligns entries whenever any non-physical/adhoc column is present. That can map a label to the wrong column object and generate incorrect grouping SQL. Build the mapping from the original columns and their actual labels in the same pass (without dropping entries first) so each label always points to its correct column definition.
Validate the correctness of the flagged issue. If correct, How can I resolve this? If you propose a fix, implement it and please make it concise.
Once fix is implemented, also check other comments on the same PR, and ask user if the user wants to fix the rest of the comments as well. if said yes, then fetch all the comments validate the correctness and implement a minimal fix
```
</details>
<a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F41184&comment_hash=27e913088b389ae6e646cd1e83646c7f18305447e930343c9fba97c4efcccbfb&reaction=like'>👍</a> | <a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F41184&comment_hash=27e913088b389ae6e646cd1e83646c7f18305447e930343c9fba97c4efcccbfb&reaction=dislike'>👎</a>
##########
superset/common/query_context_processor.py:
##########
@@ -252,9 +254,54 @@ def get_query_result(self, query_object: QueryObject) -> QueryResult:
This method delegates to the datasource's get_query_result method,
which handles query execution, normalization, time offsets, and
post-processing.
+
+ When the query requests rollup ``grouping_sets`` but the engine does not
+ support native ``GROUPING SETS``, fall back to one query per level and
+ concatenate the results with ``GROUPING()``-equivalent markers, so the
+ combined result matches the shape the native path produces (SIP.md,
+ phase 3b). Engines that support it run the single native query.
"""
+ if query_object.grouping_sets and not self._supports_grouping_sets():
+ return self._grouping_sets_fallback(query_object)
return self._qc_datasource.get_query_result(query_object)
+ def _supports_grouping_sets(self) -> bool:
+ engine_spec = getattr(self._qc_datasource, "db_engine_spec", None)
+ return bool(engine_spec and engine_spec.supports_grouping_sets)
+
+ def _grouping_sets_fallback(self, query_object: QueryObject) -> QueryResult:
+ """
+ Emulate a GROUPING SETS query on engines without native support: run one
+ query per rollup level and concatenate, tagging each level's rows with
+ the same per-column markers the native path emits.
+ """
+ levels = query_object.grouping_sets
+ all_labels = get_column_names_from_columns(query_object.columns)
+ label_to_column = dict(zip(all_labels, query_object.columns, strict=False))
+
+ frames: list[pd.DataFrame] = []
+ result: QueryResult | None = None
+ for level in levels:
+ level_labels = set(level)
+ sub_query = copy.copy(query_object)
+ sub_query.grouping_sets = []
+ sub_query.columns = [
+ label_to_column[label] for label in all_labels if label in level_labels
+ ]
Review Comment:
**Suggestion:** This fallback only keeps columns found in `all_labels`, but `all_labels` is built from physical columns only; adhoc groupby columns (including pivot temporal BASE_AXIS columns sent as adhoc SQL columns) are dropped from every fallback subquery. On engines without native grouping sets, those dimensions are silently omitted, so subtotals/grand totals are computed at the wrong granularity. Include adhoc-labeled groupby columns when selecting each level so fallback results match native behavior. [logic error]
<details>
<summary><b>Severity Level:</b> Critical 🚨</summary>
```mdx
- ❌ Adhoc groupby dimensions dropped from fallback rollup queries.
- ⚠️ Totals computed at coarser-than-intended granularity levels.
```
</details>
<details>
<summary><b>Steps of Reproduction ✅ </b></summary>
```mdx
1. Configure a chart payload for the `api/v1/chart/data` endpoint that uses
`grouping_sets` with both physical and adhoc groupby dimensions, e.g. for a
pivot-table-like query: `columns = [{"expressionType": "SQL", "sqlExpression":
"DATE_TRUNC('month', dttm)", "label": "month"}, "state"]` and `grouping_sets = [["month",
"state"], ["month"], []]`, following the grouping-sets schema documented in
`superset/charts/schemas.py:32-42`.
2. Load this payload via `ChartDataQueryContextSchema().load(payload)` (patterned after
`_result_df` in `tests/integration_tests/non_additive_totals_tests.py:78-85`), which
creates a `QueryContext` and `QueryObject`. The `QueryObject.columns` list contains both
the adhoc SQL column and the physical `state` column
(`superset/common/query_object.py:114-162`).
3. Execute the query on a database engine where `db_engine_spec.supports_grouping_sets` is
false (e.g. SQLite, noted in `tests/integration_tests/non_additive_totals_tests.py:4-10`).
`QueryContext.get_query_result()` (`superset/common/query_context.py:134-135`) invokes
`QueryContextProcessor.get_query_result()`
(`superset/common/query_context_processor.py:250-266`), which detects `grouping_sets` and
routes to `_grouping_sets_fallback()` for non-supporting engines.
4. In `_grouping_sets_fallback()` (`superset/common/query_context_processor.py:272-290`),
`all_labels = get_column_names_from_columns(query_object.columns)` calls the helper
defined in `superset/utils/core.py:1699-1707`, which expressly ignores adhoc columns. The
per-level column selection `sub_query.columns = [label_to_column[label] for label in
all_labels if label in level_labels]` therefore only ever includes physical labels from
`all_labels`: adhoc groupby dimensions present in `query_object.grouping_sets` are never
added to `sub_query.columns`. On fallback engines, rollup queries run at a reduced
grouping granularity (e.g. grouped only by `state` instead of `[month, state]`), so
subtotals and grand totals for non-additive metrics are computed over the wrong dimension
set compared to the native GROUPING SETS path.
```
</details>
[](https://app.codeant.ai/fix-in-ide?tool=cursor&prompt_id=33eb2f9e225647bbbcf800dafc731b9d&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset) [](https://app.codeant.ai/fix-in-ide?tool=vscode-claude&prompt_id=33eb2f9e225647bbbcf800dafc731b9d&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset)
*(Use Cmd/Ctrl + Click for best experience)*
<details>
<summary><b>Prompt for AI Agent 🤖 </b></summary>
```mdx
This is a comment left during a code review.
**Path:** superset/common/query_context_processor.py
**Line:** 288:290
**Comment:**
*Logic Error: This fallback only keeps columns found in `all_labels`, but `all_labels` is built from physical columns only; adhoc groupby columns (including pivot temporal BASE_AXIS columns sent as adhoc SQL columns) are dropped from every fallback subquery. On engines without native grouping sets, those dimensions are silently omitted, so subtotals/grand totals are computed at the wrong granularity. Include adhoc-labeled groupby columns when selecting each level so fallback results match native behavior.
Validate the correctness of the flagged issue. If correct, How can I resolve this? If you propose a fix, implement it and please make it concise.
Once fix is implemented, also check other comments on the same PR, and ask user if the user wants to fix the rest of the comments as well. if said yes, then fetch all the comments validate the correctness and implement a minimal fix
```
</details>
<a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F41184&comment_hash=62efd414b0b8c10bb055ea396a7a9695fd4aa167766521755e9f8f5891017404&reaction=like'>👍</a> | <a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F41184&comment_hash=62efd414b0b8c10bb055ea396a7a9695fd4aa167766521755e9f8f5891017404&reaction=dislike'>👎</a>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(table/pivot-table): correct non-additive totals/subtotals via DB rollup [POC for SIP-216] [superset]
Posted by "rusackas (via GitHub)" <gi...@apache.org>.
rusackas commented on code in PR #41184:
URL: https://github.com/apache/superset/pull/41184#discussion_r3565628555
##########
superset/common/grouping_sets.py:
##########
@@ -0,0 +1,117 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""
+SQL building blocks for the pivot-table non-additive totals optimization
+(SIP.md, phase 3b). When a datasource engine reports
+``supports_grouping_sets``, the N per-rollup-level queries can be collapsed into
+a single ``GROUPING SETS`` query: the database computes every level in one scan,
+and each returned row is attributed to its level via ``GROUPING()`` markers.
+
+These are the engine-agnostic SQL primitives. Wiring them into the query context
+(emitting one query and splitting the result back into per-level results) is the
+remaining integration; see SIP.md.
+"""
+
+from __future__ import annotations
+
+from collections.abc import Sequence
+from typing import Final
+
+import pandas as pd
+from sqlalchemy import func, tuple_
+from sqlalchemy.sql.elements import ColumnElement
+
+# Suffix for the per-column GROUPING() marker columns added to a GROUPING SETS
+# query. Chosen to be unlikely to collide with a real metric/column label.
+GROUPING_MARKER_SUFFIX: Final = "__superset_grouping"
+
+
+def grouping_marker_label(column_label: str) -> str:
+ """The output label of the GROUPING() marker for a groupby column."""
+ return f"{column_label}{GROUPING_MARKER_SUFFIX}"
+
+
+def grouping_sets_clause(
+ groups: Sequence[Sequence[ColumnElement]],
+) -> ColumnElement:
+ """
+ Build a ``GROUP BY GROUPING SETS (...)`` clause from rollup column groups.
+
+ Each group is the set of columns grouped at one rollup level; the empty
+ group ``()`` is the grand total. For example, groups ``[[a, b], [a], []]``
+ produce ``GROUPING SETS ((a, b), (a), ())``.
+
+ :param groups: one column list per rollup level
+ :return: a clause element suitable for ``select(...).group_by(...)``
+ """
+ return func.grouping_sets(*[tuple_(*group) for group in groups])
+
+
+def grouping_id_column(column: ColumnElement, label: str) -> ColumnElement:
+ """
+ Build a ``GROUPING(col) AS label`` marker column.
+
+ In a ``GROUPING SETS`` result, ``GROUPING(col)`` is ``0`` when ``col`` is
+ part of the row's grouping level and ``1`` when it has been rolled up
+ (aggregated away). Selecting one marker per groupby column lets the caller
+ attribute each returned row to its rollup level when splitting the single
+ result back into per-level results.
+
+ :param column: the groupby column to probe
+ :param label: the output label for the marker (see ``grouping_marker_label``)
+ :return: the labelled ``GROUPING(col)`` column
+ """
+ return func.grouping(column).label(label)
+
+
+def split_grouping_sets_result(
+ df: pd.DataFrame,
+ levels: Sequence[Sequence[str]],
+ groupby_columns: Sequence[str],
+) -> list[pd.DataFrame]:
+ """
+ Split a combined ``GROUPING SETS`` result into one DataFrame per rollup
+ level, the inverse of {@link grouping_sets_clause}.
Review Comment:
Fixed, swapped it for a proper Sphinx `:func:` reference.
##########
superset-frontend/plugins/plugin-chart-pivot-table/src/PivotTableChart.tsx:
##########
@@ -383,22 +338,45 @@ export default function PivotTableChart(props: PivotTableProps) {
const unpivotedData = useMemo(
() =>
- data.reduce(
- (acc: DataRecord[], record: DataRecord) => [
- ...acc,
- ...metricNames
+ // `data` is now one entry per rollup level. Tag every record with the
+ // row/column dimension labels of the level that produced it (mirroring
+ // the METRIC_KEY injection used for the full rows/cols below) so
+ // PivotData can slot each pre-computed value without re-aggregating.
+ // buildGroupbyCombinations already applied transposePivot, so the level's
+ // groupby is display-oriented and is not transposed again here.
+ data.flatMap((query: QueryData) => {
+ let levelRows = query.groupby.rows.map(getColumnLabel);
+ let levelCols = query.groupby.columns.map(getColumnLabel);
+ if (metricsLayout === MetricsLayoutEnum.ROWS) {
+ levelRows = combineMetric
+ ? [...levelRows, METRIC_KEY]
+ : [METRIC_KEY, ...levelRows];
+ } else {
+ levelCols = combineMetric
+ ? [...levelCols, METRIC_KEY]
+ : [METRIC_KEY, ...levelCols];
+ }
+ return query.data.flatMap((record: DataRecord) =>
+ metricNames
.map((name: string) => ({
...record,
[METRIC_KEY]: name,
value: record[name],
// Mark currency column for per-cell currency detection in aggregators
__currencyColumn: currencyCodeColumn,
+ // The level this record belongs to (used by PivotData placement).
+ // Namespaced with a `__` prefix (like `__metricKey` below) so it
+ // can't collide with a real dataset column named `rows`/`columns`.
+ __rows: levelRows,
+ __columns: levelCols,
+ // Identify the metric pseudo-dimension so PivotData can feed the
+ // metric-collapsed totals (the opposite "Total" axis + corner).
+ __metricKey: METRIC_KEY,
}))
- .filter(record => record.value !== null),
- ],
- [],
- ),
- [data, metricNames, currencyCodeColumn],
+ .filter(r => r.value !== null),
Review Comment:
Already fixed in the latest push, same finding as the thread above.
##########
superset-frontend/plugins/plugin-chart-pivot-table/src/plugin/transformProps.ts:
##########
@@ -88,20 +98,63 @@ export default function transformProps(chartProps: ChartProps<QueryFormData>) {
emitCrossFilters,
theme,
} = chartProps;
- const {
- data,
- colnames,
- coltypes,
- detected_currency: detectedCurrency,
- } = queriesData[0];
+ const groupbyCombinations = buildGroupbyCombinations(
+ formData as PivotTableQueryFormData,
+ );
+ const metricsArr = ensureIsArray(formData.metrics);
+ let data: QueryData[];
+ if (allMetricsAdditive(metricsArr)) {
+ // Additive fast-path: a single full-detail query was issued; synthesize
+ // each rollup level by reducing the leaf rows on the client (see SIP.md).
+ const leafRows = queriesData[0].data;
+ const metricReducers: Record<string, RollupReducer> = {};
+ metricsArr.forEach(metric => {
+ metricReducers[getMetricLabel(metric)] = additiveReducerFor(metric);
+ });
+ const labelLevels = groupbyCombinations.map(combination => ({
+ rows: combination.rows.map(getColumnLabel),
+ columns: combination.columns.map(getColumnLabel),
+ }));
+ const synthesized = synthesizeAdditiveLevels(
+ leafRows,
+ labelLevels,
+ metricReducers,
+ );
Review Comment:
Good catch, `MIN`/`MAX` were getting force-coerced through `Number()` which drops temporal/string values to null. Fixed by comparing raw values for min/max and only coercing for `sum`.
##########
superset-frontend/plugins/plugin-chart-pivot-table/src/plugin/buildQuery.ts:
##########
@@ -54,6 +59,39 @@ export default function buildQuery(formData: PivotTableQueryFormData) {
}
return col;
});
+}
+
+export default function buildQuery(formData: PivotTableQueryFormData) {
+ const { extra_form_data } = formData;
+ const time_grain_sqla =
+ extra_form_data?.time_grain_sqla || formData.time_grain_sqla;
Review Comment:
That's not right, already populates from by default. The here is only used to tag the BASE_AXIS column, and there is already a passing test () covering the override.
##########
superset-frontend/plugins/plugin-chart-pivot-table/test/react-pivottable/tableRenders.test.tsx:
##########
@@ -149,18 +202,83 @@ test('TableRenderer renders col totals row when colTotals is enabled', () => {
test('TableRenderer renders grand total when both totals are enabled', () => {
const props = buildDefaultProps({
+ data: TAGGED_COUNT_DATA,
+ vals: ['value'],
tableOptions: { rowTotals: true, colTotals: true },
});
renderWithTheme(<TableRenderer {...props} />);
- // The grand total cell should show "4" (total record count).
+ // The grand total cell shows the DB-computed "4" (total record count).
const grandTotalCells = screen
.getAllByRole('gridcell')
.filter(cell => cell.classList.contains('pvtGrandTotal'));
expect(grandTotalCells.length).toBe(1);
expect(grandTotalCells[0]).toHaveTextContent('4');
Review Comment:
does a substring match by default, so satisfies . Test passes as written.
##########
superset-frontend/plugins/plugin-chart-pivot-table/src/react-pivottable/utilities.ts:
##########
@@ -1088,76 +1117,108 @@ class PivotData {
}
processRecord(record: PivotRecord): void {
- // this code is called in a tight loop
+ // this code is called in a tight loop.
+ // Each record is tagged (in PivotTableChart) with `__rows`/`__columns`:
+ // the dimension labels of the rollup level that produced it. The database
+ // has already aggregated that level, so we place the value into exactly
+ // one slot and store it verbatim (no re-aggregation). A key shorter than
+ // the full dimension list identifies a subtotal level. Records without
+ // tags (e.g. direct unit-test construction) fall back to the full
+ // dimension lists, i.e. they are treated as leaf rows. The `__` prefix
+ // keeps these rollup tags from colliding with a real dataset column
+ // named `rows`/`columns`.
+ const recordRows = (record.__rows as unknown as string[]) ?? null;
+ const recordColumns = (record.__columns as unknown as string[]) ?? null;
+ const levelRows = recordRows ?? (this.props.rows as string[]);
+ const levelColumns = recordColumns ?? (this.props.cols as string[]);
+
const colKey: string[] = [];
const rowKey: string[] = [];
- (this.props.cols as string[]).forEach((col: string) => {
+ levelColumns.forEach((col: string) => {
colKey.push(col in record ? String(record[col]) : 'null');
});
- (this.props.rows as string[]).forEach((row: string) => {
+ levelRows.forEach((row: string) => {
rowKey.push(row in record ? String(record[row]) : 'null');
});
- this.allTotal.push(record);
-
- const rowStart = this.subtotals.rowEnabled ? 1 : Math.max(1, rowKey.length);
- const colStart = this.subtotals.colEnabled ? 1 : Math.max(1, colKey.length);
-
- let isRowSubtotal;
- let isColSubtotal;
- for (let ri = rowStart; ri <= rowKey.length; ri += 1) {
- isRowSubtotal = ri < rowKey.length;
- const fRowKey = rowKey.slice(0, ri);
- const flatRowKey = flatKey(fRowKey);
- if (!this.rowTotals[flatRowKey]) {
- this.rowKeys.push(fRowKey);
- this.rowTotals[flatRowKey] = this.getFormattedAggregator(
- record,
- rowKey,
- )(this, fRowKey, []);
+ const flatRowKey = flatKey(rowKey);
+ const flatColKey = flatKey(colKey);
+
+ const isColSubtotal = colKey.length < (this.props.cols as string[]).length;
+ const isRowSubtotal = rowKey.length < (this.props.rows as string[]).length;
+
+ // Register/create the column-total slot the first time this colKey is seen.
+ if (colKey.length > 0 && !this.colTotals[flatColKey]) {
+ if (!isColSubtotal || this.subtotals.colEnabled) {
+ this.colKeys.push(colKey);
}
- this.rowTotals[flatRowKey].push(record);
- this.rowTotals[flatRowKey].isSubtotal = isRowSubtotal;
+ this.colTotals[flatColKey] = this.getFormattedAggregator(record, colKey)(
+ this,
+ [],
+ colKey,
+ );
}
-
- for (let ci = colStart; ci <= colKey.length; ci += 1) {
- isColSubtotal = ci < colKey.length;
- const fColKey = colKey.slice(0, ci);
- const flatColKey = flatKey(fColKey);
- if (!this.colTotals[flatColKey]) {
- this.colKeys.push(fColKey);
- this.colTotals[flatColKey] = this.getFormattedAggregator(
- record,
+ // Register/create the row-total slot the first time this rowKey is seen.
+ if (rowKey.length > 0 && !this.rowTotals[flatRowKey]) {
+ if (!isRowSubtotal || this.subtotals.rowEnabled) {
+ this.rowKeys.push(rowKey);
+ }
+ this.rowTotals[flatRowKey] = this.getFormattedAggregator(record, rowKey)(
+ this,
+ rowKey,
+ [],
+ );
+ }
+ // Create the body-cell slot.
+ if (rowKey.length > 0 && colKey.length > 0) {
+ if (!this.tree[flatRowKey]) {
+ this.tree[flatRowKey] = {};
+ }
+ if (!this.tree[flatRowKey][flatColKey]) {
+ this.tree[flatRowKey][flatColKey] = this.getFormattedAggregator(record)(
+ this,
+ rowKey,
colKey,
- )(this, [], fColKey);
+ );
}
+ }
+
+ // Place the value in exactly one slot, determined by the level.
+ if (rowKey.length === 0 && colKey.length === 0) {
+ this.allTotal.push(record);
+ } else if (rowKey.length === 0) {
this.colTotals[flatColKey].push(record);
this.colTotals[flatColKey].isSubtotal = isColSubtotal;
+ } else if (colKey.length === 0) {
+ this.rowTotals[flatRowKey].push(record);
+ this.rowTotals[flatRowKey].isSubtotal = isRowSubtotal;
+ } else {
+ this.tree[flatRowKey][flatColKey].push(record);
+ this.tree[flatRowKey][flatColKey].isRowSubtotal = isRowSubtotal;
+ this.tree[flatRowKey][flatColKey].isColSubtotal = isColSubtotal;
+ this.tree[flatRowKey][flatColKey].isSubtotal =
+ isRowSubtotal || isColSubtotal;
}
- // And now fill in for all the sub-cells.
- for (let ri = rowStart; ri <= rowKey.length; ri += 1) {
- isRowSubtotal = ri < rowKey.length;
- const fRowKey = rowKey.slice(0, ri);
- const flatRowKey = flatKey(fRowKey);
- if (!this.tree[flatRowKey]) {
- this.tree[flatRowKey] = {};
+ // Metric-collapse totals. The metric is a pseudo-dimension always present on
+ // one axis, so no rollup level produces an empty key on that axis -- which
+ // would leave the opposite "Total" axis and the grand-total corner empty.
+ // When a record's axis holds only the metric (no real dims there), its value
+ // is also the collapsed total for that axis, so mirror it into rowTotals /
+ // colTotals / allTotal. (For a single metric this equals the metric column;
+ // for multiple metrics it is the last metric -- a cross-metric total is not
+ // well defined and is left as future work.)
+ const metricKey = record.__metricKey as unknown as string | undefined;
+ if (metricKey) {
+ const realColCount = levelColumns.filter(c => c !== metricKey).length;
+ const realRowCount = levelRows.filter(r => r !== metricKey).length;
+ if (levelColumns.includes(metricKey) && realColCount === 0) {
+ if (rowKey.length === 0) this.allTotal.push(record);
+ else this.rowTotals[flatRowKey]?.push(record);
}
- for (let ci = colStart; ci <= colKey.length; ci += 1) {
- isColSubtotal = ci < colKey.length;
- const fColKey = colKey.slice(0, ci);
- const flatColKey = flatKey(fColKey);
- if (!this.tree[flatRowKey][flatColKey]) {
- this.tree[flatRowKey][flatColKey] = this.getFormattedAggregator(
- record,
- )(this, fRowKey, fColKey);
- }
- this.tree[flatRowKey][flatColKey].push(record);
-
- this.tree[flatRowKey][flatColKey].isRowSubtotal = isRowSubtotal;
- this.tree[flatRowKey][flatColKey].isColSubtotal = isColSubtotal;
- this.tree[flatRowKey][flatColKey].isSubtotal =
- isRowSubtotal || isColSubtotal;
+ if (levelRows.includes(metricKey) && realRowCount === 0) {
+ if (colKey.length === 0) this.allTotal.push(record);
+ else this.colTotals[flatColKey]?.push(record);
}
Review Comment:
Different toggle: / gate intermediate subtotal levels, not the fully-collapsed totals level this mirrors into. When / are off, never requests that level's query in the first place, so never sees these records to begin with.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(table/pivot-table): correct non-additive totals/subtotals via DB rollup [SIP-216] [superset]
Posted by "bito-code-review[bot] (via GitHub)" <gi...@apache.org>.
bito-code-review[bot] commented on PR #41184:
URL: https://github.com/apache/superset/pull/41184#issuecomment-5041501312
AI Code Review is in progress (usually takes 3 to 15 minutes unless it's a very large PR).
<details>
<summary>
Bito Usage Guide
</summary>
**Commands**
Type the following command in the pull request comment and save the comment.
- `/review` - Manually triggers a full AI review.
- `/pause` - Pauses automatic reviews on this pull request.
- `/resume` - Resumes automatic reviews.
- `/resolve` - Marks all Bito-posted review comments as resolved.
- `/abort` - Cancels all in-progress reviews.
Refer to the <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/available-commands" target="_blank" rel="noopener noreferrer">documentation</a> for additional commands.
**Configuration**
This repository uses `Superset` You can customize the agent settings <a href="https://alpha.bito.ai/home/ai-agents/code-review-agent" target="_blank" rel="noopener noreferrer">here</a> or contact your Bito workspace admin at evan@preset.io.
**Documentation & Help**
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/getting-started/install-run-using-bito-cloud/create-or-customize-an-agent-instance" target="_blank" rel="noopener noreferrer">Customize agent settings</a>
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/implementing-custom-code-review-rules" target="_blank" rel="noopener noreferrer">Review rules</a>
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent" target="_blank" rel="noopener noreferrer">General documentation</a>
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/faqs" target="_blank" rel="noopener noreferrer">FAQ</a>
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(table/pivot-table): correct non-additive totals/subtotals via DB rollup [POC for SIP-216] [superset]
Posted by "codeant-ai-for-open-source[bot] (via GitHub)" <gi...@apache.org>.
codeant-ai-for-open-source[bot] commented on code in PR #41184:
URL: https://github.com/apache/superset/pull/41184#discussion_r3555846271
##########
superset-frontend/plugins/plugin-chart-pivot-table/test/plugin/transformProps.test.ts:
##########
@@ -64,36 +63,73 @@ const chartProps = new ChartProps<QueryFormData>({
theme: supersetTheme,
});
-test('should transform chart props for viz', () => {
- expect(transformProps(chartProps)).toEqual({
- width: 800,
- height: 600,
- groupbyRows: ['row1', 'row2'],
- groupbyColumns: ['col1', 'col2'],
- metrics: ['metric1', 'metric2'],
- tableRenderer: 'Table With Subtotal',
- colOrder: 'key_a_to_z',
- rowOrder: 'key_a_to_z',
- aggregateFunction: 'Sum',
- transposePivot: true,
- combineMetric: true,
- rowSubtotalPosition: true,
- colSubtotalPosition: true,
+test('should pass through formData props for viz', () => {
+ const result = transformProps(chartProps) as any;
+ expect(result.width).toBe(800);
+ expect(result.height).toBe(600);
+ expect(result.groupbyRows).toEqual(['row1', 'row2']);
+ expect(result.groupbyColumns).toEqual(['col1', 'col2']);
+ expect(result.metrics).toEqual(['metric1', 'metric2']);
+ expect(result.metricsLayout).toBe(MetricsLayoutEnum.COLUMNS);
+ expect(result.currencyFormat).toEqual({
+ symbol: 'USD',
+ symbolPosition: 'prefix',
+ });
+ // data is the per-level QueryData[] (split/synthesized), not raw rows.
+ expect(Array.isArray(result.data)).toBe(true);
+ result.data.forEach((level: any) => {
Review Comment:
**Suggestion:** Type the callback parameter with the appropriate level/query-data interface instead of `any`. [custom_rule]
**Severity Level:** Minor ⚠️
<details>
<summary><b>Why it matters? 🤔 </b></summary>
The callback parameter is explicitly typed as `any` in the changed test code. That is a direct violation of the rule against `any` in new or changed TypeScript code.
</details>
<details>
<summary><b>Rule source 📖 </b></summary>
.cursor/rules/dev-standard.mdc (line 16)
</details>
[](https://app.codeant.ai/fix-in-ide?tool=cursor&prompt_id=1c274faafa0f4087954bc9846361baf4&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset) [](https://app.codeant.ai/fix-in-ide?tool=vscode-claude&prompt_id=1c274faafa0f4087954bc9846361baf4&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset)
*(Use Cmd/Ctrl + Click for best experience)*
<details>
<summary><b>Prompt for AI Agent 🤖 </b></summary>
```mdx
This is a comment left during a code review.
**Path:** superset-frontend/plugins/plugin-chart-pivot-table/test/plugin/transformProps.test.ts
**Line:** 80:80
**Comment:**
*Custom Rule: Type the callback parameter with the appropriate level/query-data interface instead of `any`.
Validate the correctness of the flagged issue. If correct, How can I resolve this? If you propose a fix, implement it and please make it concise.
Once fix is implemented, also check other comments on the same PR, and ask user if the user wants to fix the rest of the comments as well. if said yes, then fetch all the comments validate the correctness and implement a minimal fix
```
</details>
<a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F41184&comment_hash=b3ef61008e30676a38c50270548841381d9765a448409e9419fc4dcce3d2bfb0&reaction=like'>👍</a> | <a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F41184&comment_hash=b3ef61008e30676a38c50270548841381d9765a448409e9419fc4dcce3d2bfb0&reaction=dislike'>👎</a>
##########
superset-frontend/plugins/plugin-chart-pivot-table/test/plugin/transformProps.test.ts:
##########
@@ -64,36 +63,73 @@ const chartProps = new ChartProps<QueryFormData>({
theme: supersetTheme,
});
-test('should transform chart props for viz', () => {
- expect(transformProps(chartProps)).toEqual({
- width: 800,
- height: 600,
- groupbyRows: ['row1', 'row2'],
- groupbyColumns: ['col1', 'col2'],
- metrics: ['metric1', 'metric2'],
- tableRenderer: 'Table With Subtotal',
- colOrder: 'key_a_to_z',
- rowOrder: 'key_a_to_z',
- aggregateFunction: 'Sum',
- transposePivot: true,
- combineMetric: true,
- rowSubtotalPosition: true,
- colSubtotalPosition: true,
+test('should pass through formData props for viz', () => {
+ const result = transformProps(chartProps) as any;
Review Comment:
**Suggestion:** Replace this `any` cast with the concrete transform result type so the assertions remain type-safe. [custom_rule]
**Severity Level:** Minor ⚠️
<details>
<summary><b>Why it matters? 🤔 </b></summary>
The updated TypeScript test still uses `as any` on the transform result. This directly violates the no-any-types rule, so the suggestion correctly identifies a real issue.
</details>
<details>
<summary><b>Rule source 📖 </b></summary>
.cursor/rules/dev-standard.mdc (line 16)
</details>
[](https://app.codeant.ai/fix-in-ide?tool=cursor&prompt_id=baea13d67da44e03b6b84bec65456e6b&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset) [](https://app.codeant.ai/fix-in-ide?tool=vscode-claude&prompt_id=baea13d67da44e03b6b84bec65456e6b&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset)
*(Use Cmd/Ctrl + Click for best experience)*
<details>
<summary><b>Prompt for AI Agent 🤖 </b></summary>
```mdx
This is a comment left during a code review.
**Path:** superset-frontend/plugins/plugin-chart-pivot-table/test/plugin/transformProps.test.ts
**Line:** 67:67
**Comment:**
*Custom Rule: Replace this `any` cast with the concrete transform result type so the assertions remain type-safe.
Validate the correctness of the flagged issue. If correct, How can I resolve this? If you propose a fix, implement it and please make it concise.
Once fix is implemented, also check other comments on the same PR, and ask user if the user wants to fix the rest of the comments as well. if said yes, then fetch all the comments validate the correctness and implement a minimal fix
```
</details>
<a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F41184&comment_hash=6ada9c9e6dba403279bfbee20cf5146b070d205d8b26d4e263297f9e0c48784f&reaction=like'>👍</a> | <a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F41184&comment_hash=6ada9c9e6dba403279bfbee20cf5146b070d205d8b26d4e263297f9e0c48784f&reaction=dislike'>👎</a>
##########
superset-frontend/plugins/plugin-chart-pivot-table/test/plugin/transformProps.test.ts:
##########
@@ -64,36 +63,73 @@ const chartProps = new ChartProps<QueryFormData>({
theme: supersetTheme,
});
-test('should transform chart props for viz', () => {
- expect(transformProps(chartProps)).toEqual({
- width: 800,
- height: 600,
- groupbyRows: ['row1', 'row2'],
- groupbyColumns: ['col1', 'col2'],
- metrics: ['metric1', 'metric2'],
- tableRenderer: 'Table With Subtotal',
- colOrder: 'key_a_to_z',
- rowOrder: 'key_a_to_z',
- aggregateFunction: 'Sum',
- transposePivot: true,
- combineMetric: true,
- rowSubtotalPosition: true,
- colSubtotalPosition: true,
+test('should pass through formData props for viz', () => {
+ const result = transformProps(chartProps) as any;
+ expect(result.width).toBe(800);
+ expect(result.height).toBe(600);
+ expect(result.groupbyRows).toEqual(['row1', 'row2']);
+ expect(result.groupbyColumns).toEqual(['col1', 'col2']);
+ expect(result.metrics).toEqual(['metric1', 'metric2']);
+ expect(result.metricsLayout).toBe(MetricsLayoutEnum.COLUMNS);
+ expect(result.currencyFormat).toEqual({
+ symbol: 'USD',
+ symbolPosition: 'prefix',
+ });
+ // data is the per-level QueryData[] (split/synthesized), not raw rows.
+ expect(Array.isArray(result.data)).toBe(true);
+ result.data.forEach((level: any) => {
+ expect(level).toHaveProperty('groupby');
+ expect(level).toHaveProperty('data');
+ });
+});
+
+test('non-additive: transformProps splits the GROUPING SETS result by level', () => {
+ const gm = (col: string) => `${col}__superset_grouping`;
+ const localFormData = {
+ ...formData,
+ combineMetric: false,
+ transposePivot: false,
+ metricsLayout: MetricsLayoutEnum.ROWS,
+ groupbyRows: ['region'],
+ groupbyColumns: [],
colTotals: true,
rowTotals: true,
- valueFormat: 'SMART_NUMBER',
- data: [{ name: 'Hulk', sum__num: 1, __timestamp: 599616000000 }],
- setDataMask,
- selectedFilters: {},
- verboseMap: {},
- metricsLayout: MetricsLayoutEnum.COLUMNS,
- metricColorFormatters: [],
- dateFormatters: {},
- emitCrossFilters: false,
- columnFormats: {},
- currencyFormats: {},
- currencyFormat: { symbol: 'USD', symbolPosition: 'prefix' },
+ metrics: ['m'], // saved-metric string -> non-additive
+ };
+ const cp = new ChartProps<QueryFormData>({
+ formData: localFormData as unknown as QueryFormData,
+ width: 800,
+ height: 600,
+ queriesData: [
+ {
+ // One combined GROUPING SETS result: leaf rows (region marker 0) +
+ // grand total row (region marker 1).
+ data: [
+ { region: 'US', m: 10, [gm('region')]: 0 },
+ { region: 'EU', m: 5, [gm('region')]: 0 },
+ { region: null, m: 15, [gm('region')]: 1 },
+ ],
+ colnames: ['region', 'm', gm('region')],
+ coltypes: [1, 0, 0],
+ },
+ ],
+ hooks: { setDataMask },
+ filterState: { selectedFilters: {} },
+ datasource: { verboseMap: {}, columnFormats: {} },
+ theme: supersetTheme,
});
+
+ const result = transformProps(cp) as any;
+ const grand = result.data.find(
+ (d: any) => d.groupby.rows.length === 0 && d.groupby.columns.length === 0,
Review Comment:
**Suggestion:** Give the predicate parameter a concrete type that includes `groupby.rows` and `groupby.columns` instead of `any`. [custom_rule]
**Severity Level:** Minor ⚠️
<details>
<summary><b>Why it matters? 🤔 </b></summary>
The predicate parameter is explicitly annotated as `any`, so the new code violates the no-any-types rule.
</details>
<details>
<summary><b>Rule source 📖 </b></summary>
.cursor/rules/dev-standard.mdc (line 16)
</details>
[](https://app.codeant.ai/fix-in-ide?tool=cursor&prompt_id=5bf4f19075b54fdeb80248f3f761ecb6&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset) [](https://app.codeant.ai/fix-in-ide?tool=vscode-claude&prompt_id=5bf4f19075b54fdeb80248f3f761ecb6&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset)
*(Use Cmd/Ctrl + Click for best experience)*
<details>
<summary><b>Prompt for AI Agent 🤖 </b></summary>
```mdx
This is a comment left during a code review.
**Path:** superset-frontend/plugins/plugin-chart-pivot-table/test/plugin/transformProps.test.ts
**Line:** 124:124
**Comment:**
*Custom Rule: Give the predicate parameter a concrete type that includes `groupby.rows` and `groupby.columns` instead of `any`.
Validate the correctness of the flagged issue. If correct, How can I resolve this? If you propose a fix, implement it and please make it concise.
Once fix is implemented, also check other comments on the same PR, and ask user if the user wants to fix the rest of the comments as well. if said yes, then fetch all the comments validate the correctness and implement a minimal fix
```
</details>
<a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F41184&comment_hash=934cb46acd166ad49f40420add1879f7c851d101e20d8e7eff60c3f9abd51ff2&reaction=like'>👍</a> | <a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F41184&comment_hash=934cb46acd166ad49f40420add1879f7c851d101e20d8e7eff60c3f9abd51ff2&reaction=dislike'>👎</a>
##########
superset-frontend/plugins/plugin-chart-pivot-table/test/plugin/utilities.test.ts:
##########
@@ -0,0 +1,446 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import { TimeGranularity } from '@superset-ui/core';
+import buildGroupbyCombinations, {
+ isAdditiveMetric,
+ allMetricsAdditive,
+ additiveReducerFor,
+ synthesizeAdditiveLevels,
+ splitGroupingSetsResult,
+ groupingMarkerLabel,
+} from '../../src/plugin/utilities';
+import { PivotTableQueryFormData, MetricsLayoutEnum } from '../../src/types';
+
+const baseFormData = {
+ groupbyRows: ['row1', 'row2'],
+ groupbyColumns: ['col1', 'col2'],
+ metrics: ['metric1', 'metric2'],
+ tableRenderer: 'Table With Subtotal',
+ colOrder: 'key_a_to_z',
+ rowOrder: 'key_a_to_z',
+ aggregateFunction: 'Sum',
+ metricsLayout: MetricsLayoutEnum.ROWS,
+ transposePivot: false,
+ rowSubtotalPosition: true,
+ colSubtotalPosition: true,
+ colTotals: true,
+ colSubTotals: true,
+ rowTotals: true,
+ rowSubTotals: true,
+ valueFormat: 'SMART_NUMBER',
+ datasource: '5__table',
+ viz_type: 'my_chart',
+ width: 800,
+ height: 600,
+ combineMetric: false,
+ verboseMap: {},
+ columnFormats: {},
+ currencyFormats: {},
+ metricColorFormatters: [],
+ dateFormatters: {},
+ setDataMask: () => {},
+ legacy_order_by: 'count',
+ order_desc: true,
+ margin: 0,
+ time_grain_sqla: TimeGranularity.MONTH,
+ temporal_columns_lookup: { col1: true },
+ currencyFormat: { symbol: 'USD', symbolPosition: 'prefix' },
+} as unknown as PivotTableQueryFormData;
+
+test('should build all combinations for basic pivot table', () => {
+ const combinations = buildGroupbyCombinations(baseFormData);
+
+ expect(combinations).toEqual([
+ { rows: [], columns: [] },
+ { rows: [], columns: ['col1'] },
+ { rows: [], columns: ['col1', 'col2'] },
+ { rows: ['row1'], columns: [] },
+ { rows: ['row1'], columns: ['col1'] },
+ { rows: ['row1'], columns: ['col1', 'col2'] },
+ { rows: ['row1', 'row2'], columns: [] },
+ { rows: ['row1', 'row2'], columns: ['col1'] },
+ { rows: ['row1', 'row2'], columns: ['col1', 'col2'] },
+ ]);
+
+ expect(combinations).toHaveLength(9);
+});
+
+test('should handle transposed pivot correctly', () => {
+ const modifiedFormData = {
+ ...baseFormData,
+ transposePivot: true,
+ };
+
+ const combinations = buildGroupbyCombinations(modifiedFormData);
+
+ expect(combinations).toEqual([
+ { rows: [], columns: [] },
+ { rows: [], columns: ['row1'] },
+ { rows: [], columns: ['row1', 'row2'] },
+ { rows: ['col1'], columns: [] },
+ { rows: ['col1'], columns: ['row1'] },
+ { rows: ['col1'], columns: ['row1', 'row2'] },
+ { rows: ['col1', 'col2'], columns: [] },
+ { rows: ['col1', 'col2'], columns: ['row1'] },
+ { rows: ['col1', 'col2'], columns: ['row1', 'row2'] },
+ ]);
+});
+
+test('should filter combinations when combineMetric is true with ROWS layout', () => {
+ const modifiedFormData = {
+ ...baseFormData,
+ combineMetric: true,
+ metricsLayout: MetricsLayoutEnum.ROWS,
+ };
+
+ const combinations = buildGroupbyCombinations(modifiedFormData);
+
+ expect(combinations).toEqual([
+ { rows: ['row1', 'row2'], columns: [] },
+ { rows: ['row1', 'row2'], columns: ['col1'] },
+ { rows: ['row1', 'row2'], columns: ['col1', 'col2'] },
+ ]);
+
+ expect(combinations).toHaveLength(3);
+});
+
+test('should filter combinations when combineMetric is true with COLUMNS layout', () => {
+ const modifiedFormData = {
+ ...baseFormData,
+ combineMetric: true,
+ metricsLayout: MetricsLayoutEnum.COLUMNS,
+ };
+
+ const combinations = buildGroupbyCombinations(modifiedFormData);
+
+ expect(combinations).toEqual([
+ { rows: [], columns: ['col1', 'col2'] },
+ { rows: ['row1'], columns: ['col1', 'col2'] },
+ { rows: ['row1', 'row2'], columns: ['col1', 'col2'] },
+ ]);
+
+ expect(combinations).toHaveLength(3);
+});
+
+test('should handle single dimension in rows only', () => {
+ const modifiedFormData = {
+ ...baseFormData,
+ groupbyRows: ['row'],
+ groupbyColumns: [],
+ };
+
+ const combinations = buildGroupbyCombinations(modifiedFormData);
+
+ expect(combinations).toEqual([
+ { rows: [], columns: [] },
+ { rows: ['row'], columns: [] },
+ ]);
+
+ expect(combinations).toHaveLength(2);
+});
+
+test('should handle single dimension in columns only', () => {
+ const modifiedFormData = {
+ ...baseFormData,
+ groupbyRows: [],
+ groupbyColumns: ['col'],
+ };
+
+ const combinations = buildGroupbyCombinations(modifiedFormData);
+
+ expect(combinations).toEqual([
+ { rows: [], columns: [] },
+ { rows: [], columns: ['col'] },
+ ]);
+
+ expect(combinations).toHaveLength(2);
+});
+
+test('should handle empty groupby arrays', () => {
+ const modifiedFormData = {
+ ...baseFormData,
+ groupbyRows: [],
+ groupbyColumns: [],
+ };
+
+ const combinations = buildGroupbyCombinations(modifiedFormData);
+
+ expect(combinations).toEqual([{ rows: [], columns: [] }]);
+
+ expect(combinations).toHaveLength(1);
+});
+
+test('should work with combineMetric and transposed pivot', () => {
+ const modifiedFormData = {
+ ...baseFormData,
+ transposePivot: true,
+ combineMetric: true,
+ metricsLayout: MetricsLayoutEnum.COLUMNS,
+ };
+
+ const combinations = buildGroupbyCombinations(modifiedFormData);
+
+ expect(combinations).toEqual([
+ { rows: [], columns: ['row1', 'row2'] },
+ { rows: ['col1'], columns: ['row1', 'row2'] },
+ { rows: ['col1', 'col2'], columns: ['row1', 'row2'] },
+ ]);
+});
+
+test('should handle combineMetric with empty arrays correctly', () => {
+ const modifiedFormData = {
+ ...baseFormData,
+ groupbyRows: [],
+ groupbyColumns: ['col'],
+ combineMetric: true,
+ metricsLayout: MetricsLayoutEnum.ROWS,
+ };
+
+ const combinations = buildGroupbyCombinations(modifiedFormData);
+
+ expect(combinations).toEqual([
+ { rows: [], columns: [] },
+ { rows: [], columns: ['col'] },
+ ]);
+});
+
+test('should work with large number of dimensions', () => {
+ const modifiedFormData = {
+ ...baseFormData,
+ groupbyRows: ['r1', 'r2', 'r3', 'r4'],
+ groupbyColumns: ['c1', 'c2', 'c3'],
+ };
+
+ const combinations = buildGroupbyCombinations(modifiedFormData);
+
+ expect(combinations).toHaveLength(20);
+
+ expect(combinations).toContainEqual({ rows: [], columns: [] });
+ expect(combinations).toContainEqual({
+ rows: ['r1', 'r2', 'r3', 'r4'],
+ columns: ['c1', 'c2', 'c3'],
+ });
+});
+
+test('isAdditiveMetric: SIMPLE metrics with additive aggregates are additive', () => {
+ expect(
+ isAdditiveMetric({
+ expressionType: 'SIMPLE',
+ aggregate: 'SUM',
+ column: { column_name: 'num' },
+ label: 'sum_num',
+ } as any),
Review Comment:
**Suggestion:** Replace this `any` cast with the specific metric type used by `isAdditiveMetric` so the test remains type-safe. [custom_rule]
**Severity Level:** Minor ⚠️
<details>
<summary><b>Why it matters? 🤔 </b></summary>
The file is new TypeScript test code and this call site uses `as any`, which directly violates the no-any-types rule.
</details>
<details>
<summary><b>Rule source 📖 </b></summary>
.cursor/rules/dev-standard.mdc (line 16)
</details>
[](https://app.codeant.ai/fix-in-ide?tool=cursor&prompt_id=9254d1f412f34e57b5f66c54a32442e1&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset) [](https://app.codeant.ai/fix-in-ide?tool=vscode-claude&prompt_id=9254d1f412f34e57b5f66c54a32442e1&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset)
*(Use Cmd/Ctrl + Click for best experience)*
<details>
<summary><b>Prompt for AI Agent 🤖 </b></summary>
```mdx
This is a comment left during a code review.
**Path:** superset-frontend/plugins/plugin-chart-pivot-table/test/plugin/utilities.test.ts
**Line:** 242:249
**Comment:**
*Custom Rule: Replace this `any` cast with the specific metric type used by `isAdditiveMetric` so the test remains type-safe.
Validate the correctness of the flagged issue. If correct, How can I resolve this? If you propose a fix, implement it and please make it concise.
Once fix is implemented, also check other comments on the same PR, and ask user if the user wants to fix the rest of the comments as well. if said yes, then fetch all the comments validate the correctness and implement a minimal fix
```
</details>
<a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F41184&comment_hash=17c59b68dddffbef8cdea54c2b769887e8038fc2868fcae01d3cadd8f64350aa&reaction=like'>👍</a> | <a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F41184&comment_hash=17c59b68dddffbef8cdea54c2b769887e8038fc2868fcae01d3cadd8f64350aa&reaction=dislike'>👎</a>
##########
superset-frontend/plugins/plugin-chart-pivot-table/test/plugin/transformProps.test.ts:
##########
@@ -64,36 +63,73 @@ const chartProps = new ChartProps<QueryFormData>({
theme: supersetTheme,
});
-test('should transform chart props for viz', () => {
- expect(transformProps(chartProps)).toEqual({
- width: 800,
- height: 600,
- groupbyRows: ['row1', 'row2'],
- groupbyColumns: ['col1', 'col2'],
- metrics: ['metric1', 'metric2'],
- tableRenderer: 'Table With Subtotal',
- colOrder: 'key_a_to_z',
- rowOrder: 'key_a_to_z',
- aggregateFunction: 'Sum',
- transposePivot: true,
- combineMetric: true,
- rowSubtotalPosition: true,
- colSubtotalPosition: true,
+test('should pass through formData props for viz', () => {
+ const result = transformProps(chartProps) as any;
+ expect(result.width).toBe(800);
+ expect(result.height).toBe(600);
+ expect(result.groupbyRows).toEqual(['row1', 'row2']);
+ expect(result.groupbyColumns).toEqual(['col1', 'col2']);
+ expect(result.metrics).toEqual(['metric1', 'metric2']);
+ expect(result.metricsLayout).toBe(MetricsLayoutEnum.COLUMNS);
+ expect(result.currencyFormat).toEqual({
+ symbol: 'USD',
+ symbolPosition: 'prefix',
+ });
+ // data is the per-level QueryData[] (split/synthesized), not raw rows.
+ expect(Array.isArray(result.data)).toBe(true);
+ result.data.forEach((level: any) => {
+ expect(level).toHaveProperty('groupby');
+ expect(level).toHaveProperty('data');
+ });
+});
+
+test('non-additive: transformProps splits the GROUPING SETS result by level', () => {
+ const gm = (col: string) => `${col}__superset_grouping`;
+ const localFormData = {
+ ...formData,
+ combineMetric: false,
+ transposePivot: false,
+ metricsLayout: MetricsLayoutEnum.ROWS,
+ groupbyRows: ['region'],
+ groupbyColumns: [],
colTotals: true,
rowTotals: true,
- valueFormat: 'SMART_NUMBER',
- data: [{ name: 'Hulk', sum__num: 1, __timestamp: 599616000000 }],
- setDataMask,
- selectedFilters: {},
- verboseMap: {},
- metricsLayout: MetricsLayoutEnum.COLUMNS,
- metricColorFormatters: [],
- dateFormatters: {},
- emitCrossFilters: false,
- columnFormats: {},
- currencyFormats: {},
- currencyFormat: { symbol: 'USD', symbolPosition: 'prefix' },
+ metrics: ['m'], // saved-metric string -> non-additive
+ };
+ const cp = new ChartProps<QueryFormData>({
+ formData: localFormData as unknown as QueryFormData,
+ width: 800,
+ height: 600,
+ queriesData: [
+ {
+ // One combined GROUPING SETS result: leaf rows (region marker 0) +
+ // grand total row (region marker 1).
+ data: [
+ { region: 'US', m: 10, [gm('region')]: 0 },
+ { region: 'EU', m: 5, [gm('region')]: 0 },
+ { region: null, m: 15, [gm('region')]: 1 },
+ ],
+ colnames: ['region', 'm', gm('region')],
+ coltypes: [1, 0, 0],
+ },
+ ],
+ hooks: { setDataMask },
+ filterState: { selectedFilters: {} },
+ datasource: { verboseMap: {}, columnFormats: {} },
+ theme: supersetTheme,
});
+
+ const result = transformProps(cp) as any;
Review Comment:
**Suggestion:** Replace this `any` assertion with the expected typed transform output for the non-additive case. [custom_rule]
**Severity Level:** Minor ⚠️
<details>
<summary><b>Why it matters? 🤔 </b></summary>
This line uses `as any` in newly added test code, which matches the custom rule’s prohibited pattern exactly.
</details>
<details>
<summary><b>Rule source 📖 </b></summary>
.cursor/rules/dev-standard.mdc (line 16)
</details>
[](https://app.codeant.ai/fix-in-ide?tool=cursor&prompt_id=36fb8714c26742b89470a454a250b472&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset) [](https://app.codeant.ai/fix-in-ide?tool=vscode-claude&prompt_id=36fb8714c26742b89470a454a250b472&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset)
*(Use Cmd/Ctrl + Click for best experience)*
<details>
<summary><b>Prompt for AI Agent 🤖 </b></summary>
```mdx
This is a comment left during a code review.
**Path:** superset-frontend/plugins/plugin-chart-pivot-table/test/plugin/transformProps.test.ts
**Line:** 122:122
**Comment:**
*Custom Rule: Replace this `any` assertion with the expected typed transform output for the non-additive case.
Validate the correctness of the flagged issue. If correct, How can I resolve this? If you propose a fix, implement it and please make it concise.
Once fix is implemented, also check other comments on the same PR, and ask user if the user wants to fix the rest of the comments as well. if said yes, then fetch all the comments validate the correctness and implement a minimal fix
```
</details>
<a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F41184&comment_hash=108b8cd0e029b9c189333801a874c09856f80f0c6384a9c0be0143275b53160d&reaction=like'>👍</a> | <a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F41184&comment_hash=108b8cd0e029b9c189333801a874c09856f80f0c6384a9c0be0143275b53160d&reaction=dislike'>👎</a>
##########
superset-frontend/plugins/plugin-chart-pivot-table/test/plugin/utilities.test.ts:
##########
@@ -0,0 +1,446 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import { TimeGranularity } from '@superset-ui/core';
+import buildGroupbyCombinations, {
+ isAdditiveMetric,
+ allMetricsAdditive,
+ additiveReducerFor,
+ synthesizeAdditiveLevels,
+ splitGroupingSetsResult,
+ groupingMarkerLabel,
+} from '../../src/plugin/utilities';
+import { PivotTableQueryFormData, MetricsLayoutEnum } from '../../src/types';
+
+const baseFormData = {
+ groupbyRows: ['row1', 'row2'],
+ groupbyColumns: ['col1', 'col2'],
+ metrics: ['metric1', 'metric2'],
+ tableRenderer: 'Table With Subtotal',
+ colOrder: 'key_a_to_z',
+ rowOrder: 'key_a_to_z',
+ aggregateFunction: 'Sum',
+ metricsLayout: MetricsLayoutEnum.ROWS,
+ transposePivot: false,
+ rowSubtotalPosition: true,
+ colSubtotalPosition: true,
+ colTotals: true,
+ colSubTotals: true,
+ rowTotals: true,
+ rowSubTotals: true,
+ valueFormat: 'SMART_NUMBER',
+ datasource: '5__table',
+ viz_type: 'my_chart',
+ width: 800,
+ height: 600,
+ combineMetric: false,
+ verboseMap: {},
+ columnFormats: {},
+ currencyFormats: {},
+ metricColorFormatters: [],
+ dateFormatters: {},
+ setDataMask: () => {},
+ legacy_order_by: 'count',
+ order_desc: true,
+ margin: 0,
+ time_grain_sqla: TimeGranularity.MONTH,
+ temporal_columns_lookup: { col1: true },
+ currencyFormat: { symbol: 'USD', symbolPosition: 'prefix' },
+} as unknown as PivotTableQueryFormData;
+
+test('should build all combinations for basic pivot table', () => {
+ const combinations = buildGroupbyCombinations(baseFormData);
+
+ expect(combinations).toEqual([
+ { rows: [], columns: [] },
+ { rows: [], columns: ['col1'] },
+ { rows: [], columns: ['col1', 'col2'] },
+ { rows: ['row1'], columns: [] },
+ { rows: ['row1'], columns: ['col1'] },
+ { rows: ['row1'], columns: ['col1', 'col2'] },
+ { rows: ['row1', 'row2'], columns: [] },
+ { rows: ['row1', 'row2'], columns: ['col1'] },
+ { rows: ['row1', 'row2'], columns: ['col1', 'col2'] },
+ ]);
+
+ expect(combinations).toHaveLength(9);
+});
+
+test('should handle transposed pivot correctly', () => {
+ const modifiedFormData = {
+ ...baseFormData,
+ transposePivot: true,
+ };
+
+ const combinations = buildGroupbyCombinations(modifiedFormData);
+
+ expect(combinations).toEqual([
+ { rows: [], columns: [] },
+ { rows: [], columns: ['row1'] },
+ { rows: [], columns: ['row1', 'row2'] },
+ { rows: ['col1'], columns: [] },
+ { rows: ['col1'], columns: ['row1'] },
+ { rows: ['col1'], columns: ['row1', 'row2'] },
+ { rows: ['col1', 'col2'], columns: [] },
+ { rows: ['col1', 'col2'], columns: ['row1'] },
+ { rows: ['col1', 'col2'], columns: ['row1', 'row2'] },
+ ]);
+});
+
+test('should filter combinations when combineMetric is true with ROWS layout', () => {
+ const modifiedFormData = {
+ ...baseFormData,
+ combineMetric: true,
+ metricsLayout: MetricsLayoutEnum.ROWS,
+ };
+
+ const combinations = buildGroupbyCombinations(modifiedFormData);
+
+ expect(combinations).toEqual([
+ { rows: ['row1', 'row2'], columns: [] },
+ { rows: ['row1', 'row2'], columns: ['col1'] },
+ { rows: ['row1', 'row2'], columns: ['col1', 'col2'] },
+ ]);
+
+ expect(combinations).toHaveLength(3);
+});
+
+test('should filter combinations when combineMetric is true with COLUMNS layout', () => {
+ const modifiedFormData = {
+ ...baseFormData,
+ combineMetric: true,
+ metricsLayout: MetricsLayoutEnum.COLUMNS,
+ };
+
+ const combinations = buildGroupbyCombinations(modifiedFormData);
+
+ expect(combinations).toEqual([
+ { rows: [], columns: ['col1', 'col2'] },
+ { rows: ['row1'], columns: ['col1', 'col2'] },
+ { rows: ['row1', 'row2'], columns: ['col1', 'col2'] },
+ ]);
+
+ expect(combinations).toHaveLength(3);
+});
+
+test('should handle single dimension in rows only', () => {
+ const modifiedFormData = {
+ ...baseFormData,
+ groupbyRows: ['row'],
+ groupbyColumns: [],
+ };
+
+ const combinations = buildGroupbyCombinations(modifiedFormData);
+
+ expect(combinations).toEqual([
+ { rows: [], columns: [] },
+ { rows: ['row'], columns: [] },
+ ]);
+
+ expect(combinations).toHaveLength(2);
+});
+
+test('should handle single dimension in columns only', () => {
+ const modifiedFormData = {
+ ...baseFormData,
+ groupbyRows: [],
+ groupbyColumns: ['col'],
+ };
+
+ const combinations = buildGroupbyCombinations(modifiedFormData);
+
+ expect(combinations).toEqual([
+ { rows: [], columns: [] },
+ { rows: [], columns: ['col'] },
+ ]);
+
+ expect(combinations).toHaveLength(2);
+});
+
+test('should handle empty groupby arrays', () => {
+ const modifiedFormData = {
+ ...baseFormData,
+ groupbyRows: [],
+ groupbyColumns: [],
+ };
+
+ const combinations = buildGroupbyCombinations(modifiedFormData);
+
+ expect(combinations).toEqual([{ rows: [], columns: [] }]);
+
+ expect(combinations).toHaveLength(1);
+});
+
+test('should work with combineMetric and transposed pivot', () => {
+ const modifiedFormData = {
+ ...baseFormData,
+ transposePivot: true,
+ combineMetric: true,
+ metricsLayout: MetricsLayoutEnum.COLUMNS,
+ };
+
+ const combinations = buildGroupbyCombinations(modifiedFormData);
+
+ expect(combinations).toEqual([
+ { rows: [], columns: ['row1', 'row2'] },
+ { rows: ['col1'], columns: ['row1', 'row2'] },
+ { rows: ['col1', 'col2'], columns: ['row1', 'row2'] },
+ ]);
+});
+
+test('should handle combineMetric with empty arrays correctly', () => {
+ const modifiedFormData = {
+ ...baseFormData,
+ groupbyRows: [],
+ groupbyColumns: ['col'],
+ combineMetric: true,
+ metricsLayout: MetricsLayoutEnum.ROWS,
+ };
+
+ const combinations = buildGroupbyCombinations(modifiedFormData);
+
+ expect(combinations).toEqual([
+ { rows: [], columns: [] },
+ { rows: [], columns: ['col'] },
+ ]);
+});
+
+test('should work with large number of dimensions', () => {
+ const modifiedFormData = {
+ ...baseFormData,
+ groupbyRows: ['r1', 'r2', 'r3', 'r4'],
+ groupbyColumns: ['c1', 'c2', 'c3'],
+ };
+
+ const combinations = buildGroupbyCombinations(modifiedFormData);
+
+ expect(combinations).toHaveLength(20);
+
+ expect(combinations).toContainEqual({ rows: [], columns: [] });
+ expect(combinations).toContainEqual({
+ rows: ['r1', 'r2', 'r3', 'r4'],
+ columns: ['c1', 'c2', 'c3'],
+ });
+});
+
+test('isAdditiveMetric: SIMPLE metrics with additive aggregates are additive', () => {
+ expect(
+ isAdditiveMetric({
+ expressionType: 'SIMPLE',
+ aggregate: 'SUM',
+ column: { column_name: 'num' },
+ label: 'sum_num',
+ } as any),
+ ).toBe(true);
+ expect(
+ isAdditiveMetric({
+ expressionType: 'SIMPLE',
+ aggregate: 'COUNT',
+ column: { column_name: 'num' },
+ label: 'count_num',
+ } as any),
+ ).toBe(true);
+});
+
+test('isAdditiveMetric: non-additive aggregates, SQL, and saved metrics are not additive', () => {
+ expect(
+ isAdditiveMetric({
+ expressionType: 'SIMPLE',
+ aggregate: 'AVG',
+ column: { column_name: 'num' },
+ label: 'avg_num',
+ } as any),
+ ).toBe(false);
+ expect(
+ isAdditiveMetric({
+ expressionType: 'SIMPLE',
+ aggregate: 'COUNT_DISTINCT',
+ column: { column_name: 'name' },
+ label: 'distinct_names',
+ } as any),
+ ).toBe(false);
+ expect(
+ isAdditiveMetric({
+ expressionType: 'SQL',
+ sqlExpression: 'SUM(a) / SUM(b)',
+ label: 'ratio',
+ } as any),
+ ).toBe(false);
+ // saved-metric reference: aggregate unknown from form data -> non-additive
+ expect(isAdditiveMetric('count' as any)).toBe(false);
+});
+
+test('allMetricsAdditive: all additive vs any non-additive vs empty', () => {
+ const sum = {
+ expressionType: 'SIMPLE',
+ aggregate: 'SUM',
+ column: { column_name: 'a' },
+ label: 'a',
+ } as any;
+ const ratio = {
+ expressionType: 'SQL',
+ sqlExpression: 'SUM(a)/SUM(b)',
+ label: 'r',
+ } as any;
+ expect(allMetricsAdditive([sum, sum])).toBe(true);
+ expect(allMetricsAdditive([sum, ratio])).toBe(false);
+ expect(allMetricsAdditive([])).toBe(false);
+});
+
+test('pruning: with all totals/subtotals off, only the leaf level is queried', () => {
+ const combinations = buildGroupbyCombinations({
+ ...baseFormData,
+ colTotals: false,
+ rowTotals: false,
+ colSubTotals: false,
+ rowSubTotals: false,
+ });
+ expect(combinations).toEqual([
+ { rows: ['row1', 'row2'], columns: ['col1', 'col2'] },
+ ]);
+});
+
+test('pruning: totals on, subtotals off -> only full + fully-collapsed prefixes', () => {
+ const combinations = buildGroupbyCombinations({
+ ...baseFormData,
+ colTotals: true,
+ rowTotals: true,
+ colSubTotals: false,
+ rowSubTotals: false,
+ });
+ // 2 row prefixes ([], full) x 2 col prefixes ([], full) = 4
+ expect(combinations).toEqual([
+ { rows: [], columns: [] },
+ { rows: [], columns: ['col1', 'col2'] },
+ { rows: ['row1', 'row2'], columns: [] },
+ { rows: ['row1', 'row2'], columns: ['col1', 'col2'] },
+ ]);
+});
+
+test('pruning: row subtotals on, everything else off', () => {
+ const combinations = buildGroupbyCombinations({
+ ...baseFormData,
+ colTotals: false,
+ rowTotals: false,
+ colSubTotals: false,
+ rowSubTotals: true,
+ });
+ // row prefixes: intermediate [row1] + full [row1,row2]; col prefixes: full only
+ expect(combinations).toEqual([
+ { rows: ['row1'], columns: ['col1', 'col2'] },
+ { rows: ['row1', 'row2'], columns: ['col1', 'col2'] },
+ ]);
+});
+
+test('pruning: empty column dims keep the [] (leaf) level regardless of rowTotals', () => {
+ const combinations = buildGroupbyCombinations({
+ ...baseFormData,
+ groupbyColumns: [],
+ colTotals: true, // bottom total row
+ rowTotals: false,
+ colSubTotals: false,
+ rowSubTotals: false,
+ });
+ // columns is empty so [] is the leaf level (always kept); rows: [] (colTotals) + full
+ expect(combinations).toEqual([
+ { rows: [], columns: [] },
+ { rows: ['row1', 'row2'], columns: [] },
+ ]);
+});
+
+test('additiveReducerFor maps aggregates to reducers', () => {
+ const mk = (aggregate: string) =>
+ ({ expressionType: 'SIMPLE', aggregate, label: aggregate }) as any;
+ expect(additiveReducerFor(mk('SUM'))).toBe('sum');
+ expect(additiveReducerFor(mk('COUNT'))).toBe('sum');
+ expect(additiveReducerFor(mk('MIN'))).toBe('min');
+ expect(additiveReducerFor(mk('MAX'))).toBe('max');
+ expect(additiveReducerFor('saved_metric' as any)).toBe('sum');
Review Comment:
**Suggestion:** Pass a correctly typed saved-metric value to `additiveReducerFor` and remove the `any` cast. [custom_rule]
**Severity Level:** Minor ⚠️
<details>
<summary><b>Why it matters? 🤔 </b></summary>
The new code uses an explicit `any` cast in TypeScript, so this is a real violation of the custom rule.
</details>
<details>
<summary><b>Rule source 📖 </b></summary>
.cursor/rules/dev-standard.mdc (line 16)
</details>
[](https://app.codeant.ai/fix-in-ide?tool=cursor&prompt_id=7b14c04d4d62433689e868fd62595723&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset) [](https://app.codeant.ai/fix-in-ide?tool=vscode-claude&prompt_id=7b14c04d4d62433689e868fd62595723&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset)
*(Use Cmd/Ctrl + Click for best experience)*
<details>
<summary><b>Prompt for AI Agent 🤖 </b></summary>
```mdx
This is a comment left during a code review.
**Path:** superset-frontend/plugins/plugin-chart-pivot-table/test/plugin/utilities.test.ts
**Line:** 367:374
**Comment:**
*Custom Rule: Pass a correctly typed saved-metric value to `additiveReducerFor` and remove the `any` cast.
Validate the correctness of the flagged issue. If correct, How can I resolve this? If you propose a fix, implement it and please make it concise.
Once fix is implemented, also check other comments on the same PR, and ask user if the user wants to fix the rest of the comments as well. if said yes, then fetch all the comments validate the correctness and implement a minimal fix
```
</details>
<a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F41184&comment_hash=655e09c7e0870db465a61b4a8b8e15020b594e86782e5a19725388d452c54c11&reaction=like'>👍</a> | <a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F41184&comment_hash=655e09c7e0870db465a61b4a8b8e15020b594e86782e5a19725388d452c54c11&reaction=dislike'>👎</a>
##########
superset-frontend/plugins/plugin-chart-pivot-table/test/plugin/utilities.test.ts:
##########
@@ -0,0 +1,446 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import { TimeGranularity } from '@superset-ui/core';
+import buildGroupbyCombinations, {
+ isAdditiveMetric,
+ allMetricsAdditive,
+ additiveReducerFor,
+ synthesizeAdditiveLevels,
+ splitGroupingSetsResult,
+ groupingMarkerLabel,
+} from '../../src/plugin/utilities';
+import { PivotTableQueryFormData, MetricsLayoutEnum } from '../../src/types';
+
+const baseFormData = {
+ groupbyRows: ['row1', 'row2'],
+ groupbyColumns: ['col1', 'col2'],
+ metrics: ['metric1', 'metric2'],
+ tableRenderer: 'Table With Subtotal',
+ colOrder: 'key_a_to_z',
+ rowOrder: 'key_a_to_z',
+ aggregateFunction: 'Sum',
+ metricsLayout: MetricsLayoutEnum.ROWS,
+ transposePivot: false,
+ rowSubtotalPosition: true,
+ colSubtotalPosition: true,
+ colTotals: true,
+ colSubTotals: true,
+ rowTotals: true,
+ rowSubTotals: true,
+ valueFormat: 'SMART_NUMBER',
+ datasource: '5__table',
+ viz_type: 'my_chart',
+ width: 800,
+ height: 600,
+ combineMetric: false,
+ verboseMap: {},
+ columnFormats: {},
+ currencyFormats: {},
+ metricColorFormatters: [],
+ dateFormatters: {},
+ setDataMask: () => {},
+ legacy_order_by: 'count',
+ order_desc: true,
+ margin: 0,
+ time_grain_sqla: TimeGranularity.MONTH,
+ temporal_columns_lookup: { col1: true },
+ currencyFormat: { symbol: 'USD', symbolPosition: 'prefix' },
+} as unknown as PivotTableQueryFormData;
+
+test('should build all combinations for basic pivot table', () => {
+ const combinations = buildGroupbyCombinations(baseFormData);
+
+ expect(combinations).toEqual([
+ { rows: [], columns: [] },
+ { rows: [], columns: ['col1'] },
+ { rows: [], columns: ['col1', 'col2'] },
+ { rows: ['row1'], columns: [] },
+ { rows: ['row1'], columns: ['col1'] },
+ { rows: ['row1'], columns: ['col1', 'col2'] },
+ { rows: ['row1', 'row2'], columns: [] },
+ { rows: ['row1', 'row2'], columns: ['col1'] },
+ { rows: ['row1', 'row2'], columns: ['col1', 'col2'] },
+ ]);
+
+ expect(combinations).toHaveLength(9);
+});
+
+test('should handle transposed pivot correctly', () => {
+ const modifiedFormData = {
+ ...baseFormData,
+ transposePivot: true,
+ };
+
+ const combinations = buildGroupbyCombinations(modifiedFormData);
+
+ expect(combinations).toEqual([
+ { rows: [], columns: [] },
+ { rows: [], columns: ['row1'] },
+ { rows: [], columns: ['row1', 'row2'] },
+ { rows: ['col1'], columns: [] },
+ { rows: ['col1'], columns: ['row1'] },
+ { rows: ['col1'], columns: ['row1', 'row2'] },
+ { rows: ['col1', 'col2'], columns: [] },
+ { rows: ['col1', 'col2'], columns: ['row1'] },
+ { rows: ['col1', 'col2'], columns: ['row1', 'row2'] },
+ ]);
+});
+
+test('should filter combinations when combineMetric is true with ROWS layout', () => {
+ const modifiedFormData = {
+ ...baseFormData,
+ combineMetric: true,
+ metricsLayout: MetricsLayoutEnum.ROWS,
+ };
+
+ const combinations = buildGroupbyCombinations(modifiedFormData);
+
+ expect(combinations).toEqual([
+ { rows: ['row1', 'row2'], columns: [] },
+ { rows: ['row1', 'row2'], columns: ['col1'] },
+ { rows: ['row1', 'row2'], columns: ['col1', 'col2'] },
+ ]);
+
+ expect(combinations).toHaveLength(3);
+});
+
+test('should filter combinations when combineMetric is true with COLUMNS layout', () => {
+ const modifiedFormData = {
+ ...baseFormData,
+ combineMetric: true,
+ metricsLayout: MetricsLayoutEnum.COLUMNS,
+ };
+
+ const combinations = buildGroupbyCombinations(modifiedFormData);
+
+ expect(combinations).toEqual([
+ { rows: [], columns: ['col1', 'col2'] },
+ { rows: ['row1'], columns: ['col1', 'col2'] },
+ { rows: ['row1', 'row2'], columns: ['col1', 'col2'] },
+ ]);
+
+ expect(combinations).toHaveLength(3);
+});
+
+test('should handle single dimension in rows only', () => {
+ const modifiedFormData = {
+ ...baseFormData,
+ groupbyRows: ['row'],
+ groupbyColumns: [],
+ };
+
+ const combinations = buildGroupbyCombinations(modifiedFormData);
+
+ expect(combinations).toEqual([
+ { rows: [], columns: [] },
+ { rows: ['row'], columns: [] },
+ ]);
+
+ expect(combinations).toHaveLength(2);
+});
+
+test('should handle single dimension in columns only', () => {
+ const modifiedFormData = {
+ ...baseFormData,
+ groupbyRows: [],
+ groupbyColumns: ['col'],
+ };
+
+ const combinations = buildGroupbyCombinations(modifiedFormData);
+
+ expect(combinations).toEqual([
+ { rows: [], columns: [] },
+ { rows: [], columns: ['col'] },
+ ]);
+
+ expect(combinations).toHaveLength(2);
+});
+
+test('should handle empty groupby arrays', () => {
+ const modifiedFormData = {
+ ...baseFormData,
+ groupbyRows: [],
+ groupbyColumns: [],
+ };
+
+ const combinations = buildGroupbyCombinations(modifiedFormData);
+
+ expect(combinations).toEqual([{ rows: [], columns: [] }]);
+
+ expect(combinations).toHaveLength(1);
+});
+
+test('should work with combineMetric and transposed pivot', () => {
+ const modifiedFormData = {
+ ...baseFormData,
+ transposePivot: true,
+ combineMetric: true,
+ metricsLayout: MetricsLayoutEnum.COLUMNS,
+ };
+
+ const combinations = buildGroupbyCombinations(modifiedFormData);
+
+ expect(combinations).toEqual([
+ { rows: [], columns: ['row1', 'row2'] },
+ { rows: ['col1'], columns: ['row1', 'row2'] },
+ { rows: ['col1', 'col2'], columns: ['row1', 'row2'] },
+ ]);
+});
+
+test('should handle combineMetric with empty arrays correctly', () => {
+ const modifiedFormData = {
+ ...baseFormData,
+ groupbyRows: [],
+ groupbyColumns: ['col'],
+ combineMetric: true,
+ metricsLayout: MetricsLayoutEnum.ROWS,
+ };
+
+ const combinations = buildGroupbyCombinations(modifiedFormData);
+
+ expect(combinations).toEqual([
+ { rows: [], columns: [] },
+ { rows: [], columns: ['col'] },
+ ]);
+});
+
+test('should work with large number of dimensions', () => {
+ const modifiedFormData = {
+ ...baseFormData,
+ groupbyRows: ['r1', 'r2', 'r3', 'r4'],
+ groupbyColumns: ['c1', 'c2', 'c3'],
+ };
+
+ const combinations = buildGroupbyCombinations(modifiedFormData);
+
+ expect(combinations).toHaveLength(20);
+
+ expect(combinations).toContainEqual({ rows: [], columns: [] });
+ expect(combinations).toContainEqual({
+ rows: ['r1', 'r2', 'r3', 'r4'],
+ columns: ['c1', 'c2', 'c3'],
+ });
+});
+
+test('isAdditiveMetric: SIMPLE metrics with additive aggregates are additive', () => {
+ expect(
+ isAdditiveMetric({
+ expressionType: 'SIMPLE',
+ aggregate: 'SUM',
+ column: { column_name: 'num' },
+ label: 'sum_num',
+ } as any),
+ ).toBe(true);
+ expect(
+ isAdditiveMetric({
+ expressionType: 'SIMPLE',
+ aggregate: 'COUNT',
+ column: { column_name: 'num' },
+ label: 'count_num',
+ } as any),
+ ).toBe(true);
+});
+
+test('isAdditiveMetric: non-additive aggregates, SQL, and saved metrics are not additive', () => {
+ expect(
+ isAdditiveMetric({
+ expressionType: 'SIMPLE',
+ aggregate: 'AVG',
+ column: { column_name: 'num' },
+ label: 'avg_num',
+ } as any),
+ ).toBe(false);
+ expect(
+ isAdditiveMetric({
+ expressionType: 'SIMPLE',
+ aggregate: 'COUNT_DISTINCT',
+ column: { column_name: 'name' },
+ label: 'distinct_names',
+ } as any),
+ ).toBe(false);
+ expect(
+ isAdditiveMetric({
+ expressionType: 'SQL',
+ sqlExpression: 'SUM(a) / SUM(b)',
+ label: 'ratio',
+ } as any),
+ ).toBe(false);
+ // saved-metric reference: aggregate unknown from form data -> non-additive
+ expect(isAdditiveMetric('count' as any)).toBe(false);
Review Comment:
**Suggestion:** Remove the `any` cast and type this saved-metric argument with the correct union/member type accepted by `isAdditiveMetric`. [custom_rule]
**Severity Level:** Minor ⚠️
<details>
<summary><b>Why it matters? 🤔 </b></summary>
This new test code contains an explicit `any` cast, so it matches the rule forbidding `any` in changed TypeScript code.
</details>
<details>
<summary><b>Rule source 📖 </b></summary>
.cursor/rules/dev-standard.mdc (line 16)
</details>
[](https://app.codeant.ai/fix-in-ide?tool=cursor&prompt_id=094fe6a79d264f23ba45025a6e9737a7&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset) [](https://app.codeant.ai/fix-in-ide?tool=vscode-claude&prompt_id=094fe6a79d264f23ba45025a6e9737a7&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset)
*(Use Cmd/Ctrl + Click for best experience)*
<details>
<summary><b>Prompt for AI Agent 🤖 </b></summary>
```mdx
This is a comment left during a code review.
**Path:** superset-frontend/plugins/plugin-chart-pivot-table/test/plugin/utilities.test.ts
**Line:** 286:286
**Comment:**
*Custom Rule: Remove the `any` cast and type this saved-metric argument with the correct union/member type accepted by `isAdditiveMetric`.
Validate the correctness of the flagged issue. If correct, How can I resolve this? If you propose a fix, implement it and please make it concise.
Once fix is implemented, also check other comments on the same PR, and ask user if the user wants to fix the rest of the comments as well. if said yes, then fetch all the comments validate the correctness and implement a minimal fix
```
</details>
<a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F41184&comment_hash=12a741a48e9e04ca7116848399d636959db15e62d7d0005184d2d532128286e6&reaction=like'>👍</a> | <a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F41184&comment_hash=12a741a48e9e04ca7116848399d636959db15e62d7d0005184d2d532128286e6&reaction=dislike'>👎</a>
##########
superset-frontend/plugins/plugin-chart-pivot-table/test/plugin/transformProps.test.ts:
##########
@@ -335,3 +371,58 @@ test('should map conditional formatting rules to metricColorFormatters with corr
result.metricColorFormatters[1].getColorFromValue(column2Formatting),
).toEqual('#5ac189FF');
});
+
+test('additive metrics: synthesizes rollup levels from a single leaf query', () => {
+ const additiveFormData = {
+ ...formData,
+ combineMetric: false,
+ transposePivot: false,
+ metricsLayout: MetricsLayoutEnum.ROWS,
+ groupbyRows: ['region'],
+ groupbyColumns: [],
+ colTotals: true,
+ rowTotals: true,
+ metrics: [
+ {
+ expressionType: 'SIMPLE',
+ aggregate: 'SUM',
+ column: { column_name: 'v' },
+ label: 'v',
+ },
+ ],
+ };
+ const additiveChartProps = new ChartProps<QueryFormData>({
+ formData: additiveFormData as unknown as QueryFormData,
+ width: 800,
+ height: 600,
+ queriesData: [
+ {
+ data: [
+ { region: 'US', v: 10 },
+ { region: 'EU', v: 5 },
+ ],
+ colnames: ['region', 'v'],
+ coltypes: [1, 0],
+ },
+ ],
+ hooks: { setDataMask },
+ filterState: { selectedFilters: {} },
+ datasource: { verboseMap: {}, columnFormats: {} },
+ theme: supersetTheme,
+ });
+
+ const result = transformProps(additiveChartProps);
+ // One query produced multiple synthesized rollup levels.
+ expect(result.data.length).toBeGreaterThan(1);
+ // Grand-total level: region collapsed -> v = 10 + 5 = 15.
+ const grand = (result.data as any[]).find(
Review Comment:
**Suggestion:** Replace the `any[]` cast with a concrete array element type for rollup level entries. [custom_rule]
**Severity Level:** Minor ⚠️
<details>
<summary><b>Why it matters? 🤔 </b></summary>
The code casts `result.data` to `any[]`, which is precisely the type usage the rule forbids in changed TypeScript code.
</details>
<details>
<summary><b>Rule source 📖 </b></summary>
.cursor/rules/dev-standard.mdc (line 16)
</details>
[](https://app.codeant.ai/fix-in-ide?tool=cursor&prompt_id=d9734be3825d428aae92c67ac088858f&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset) [](https://app.codeant.ai/fix-in-ide?tool=vscode-claude&prompt_id=d9734be3825d428aae92c67ac088858f&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset)
*(Use Cmd/Ctrl + Click for best experience)*
<details>
<summary><b>Prompt for AI Agent 🤖 </b></summary>
```mdx
This is a comment left during a code review.
**Path:** superset-frontend/plugins/plugin-chart-pivot-table/test/plugin/transformProps.test.ts
**Line:** 418:418
**Comment:**
*Custom Rule: Replace the `any[]` cast with a concrete array element type for rollup level entries.
Validate the correctness of the flagged issue. If correct, How can I resolve this? If you propose a fix, implement it and please make it concise.
Once fix is implemented, also check other comments on the same PR, and ask user if the user wants to fix the rest of the comments as well. if said yes, then fetch all the comments validate the correctness and implement a minimal fix
```
</details>
<a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F41184&comment_hash=48b79180b7b6d09cdfcf9fd73cdfe520359fc6ff506dd0a06974c1a86f14d561&reaction=like'>👍</a> | <a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F41184&comment_hash=48b79180b7b6d09cdfcf9fd73cdfe520359fc6ff506dd0a06974c1a86f14d561&reaction=dislike'>👎</a>
##########
superset-frontend/plugins/plugin-chart-pivot-table/test/plugin/utilities.test.ts:
##########
@@ -0,0 +1,446 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import { TimeGranularity } from '@superset-ui/core';
+import buildGroupbyCombinations, {
+ isAdditiveMetric,
+ allMetricsAdditive,
+ additiveReducerFor,
+ synthesizeAdditiveLevels,
+ splitGroupingSetsResult,
+ groupingMarkerLabel,
+} from '../../src/plugin/utilities';
+import { PivotTableQueryFormData, MetricsLayoutEnum } from '../../src/types';
+
+const baseFormData = {
+ groupbyRows: ['row1', 'row2'],
+ groupbyColumns: ['col1', 'col2'],
+ metrics: ['metric1', 'metric2'],
+ tableRenderer: 'Table With Subtotal',
+ colOrder: 'key_a_to_z',
+ rowOrder: 'key_a_to_z',
+ aggregateFunction: 'Sum',
+ metricsLayout: MetricsLayoutEnum.ROWS,
+ transposePivot: false,
+ rowSubtotalPosition: true,
+ colSubtotalPosition: true,
+ colTotals: true,
+ colSubTotals: true,
+ rowTotals: true,
+ rowSubTotals: true,
+ valueFormat: 'SMART_NUMBER',
+ datasource: '5__table',
+ viz_type: 'my_chart',
+ width: 800,
+ height: 600,
+ combineMetric: false,
+ verboseMap: {},
+ columnFormats: {},
+ currencyFormats: {},
+ metricColorFormatters: [],
+ dateFormatters: {},
+ setDataMask: () => {},
+ legacy_order_by: 'count',
+ order_desc: true,
+ margin: 0,
+ time_grain_sqla: TimeGranularity.MONTH,
+ temporal_columns_lookup: { col1: true },
+ currencyFormat: { symbol: 'USD', symbolPosition: 'prefix' },
+} as unknown as PivotTableQueryFormData;
+
+test('should build all combinations for basic pivot table', () => {
+ const combinations = buildGroupbyCombinations(baseFormData);
+
+ expect(combinations).toEqual([
+ { rows: [], columns: [] },
+ { rows: [], columns: ['col1'] },
+ { rows: [], columns: ['col1', 'col2'] },
+ { rows: ['row1'], columns: [] },
+ { rows: ['row1'], columns: ['col1'] },
+ { rows: ['row1'], columns: ['col1', 'col2'] },
+ { rows: ['row1', 'row2'], columns: [] },
+ { rows: ['row1', 'row2'], columns: ['col1'] },
+ { rows: ['row1', 'row2'], columns: ['col1', 'col2'] },
+ ]);
+
+ expect(combinations).toHaveLength(9);
+});
+
+test('should handle transposed pivot correctly', () => {
+ const modifiedFormData = {
+ ...baseFormData,
+ transposePivot: true,
+ };
+
+ const combinations = buildGroupbyCombinations(modifiedFormData);
+
+ expect(combinations).toEqual([
+ { rows: [], columns: [] },
+ { rows: [], columns: ['row1'] },
+ { rows: [], columns: ['row1', 'row2'] },
+ { rows: ['col1'], columns: [] },
+ { rows: ['col1'], columns: ['row1'] },
+ { rows: ['col1'], columns: ['row1', 'row2'] },
+ { rows: ['col1', 'col2'], columns: [] },
+ { rows: ['col1', 'col2'], columns: ['row1'] },
+ { rows: ['col1', 'col2'], columns: ['row1', 'row2'] },
+ ]);
+});
+
+test('should filter combinations when combineMetric is true with ROWS layout', () => {
+ const modifiedFormData = {
+ ...baseFormData,
+ combineMetric: true,
+ metricsLayout: MetricsLayoutEnum.ROWS,
+ };
+
+ const combinations = buildGroupbyCombinations(modifiedFormData);
+
+ expect(combinations).toEqual([
+ { rows: ['row1', 'row2'], columns: [] },
+ { rows: ['row1', 'row2'], columns: ['col1'] },
+ { rows: ['row1', 'row2'], columns: ['col1', 'col2'] },
+ ]);
+
+ expect(combinations).toHaveLength(3);
+});
+
+test('should filter combinations when combineMetric is true with COLUMNS layout', () => {
+ const modifiedFormData = {
+ ...baseFormData,
+ combineMetric: true,
+ metricsLayout: MetricsLayoutEnum.COLUMNS,
+ };
+
+ const combinations = buildGroupbyCombinations(modifiedFormData);
+
+ expect(combinations).toEqual([
+ { rows: [], columns: ['col1', 'col2'] },
+ { rows: ['row1'], columns: ['col1', 'col2'] },
+ { rows: ['row1', 'row2'], columns: ['col1', 'col2'] },
+ ]);
+
+ expect(combinations).toHaveLength(3);
+});
+
+test('should handle single dimension in rows only', () => {
+ const modifiedFormData = {
+ ...baseFormData,
+ groupbyRows: ['row'],
+ groupbyColumns: [],
+ };
+
+ const combinations = buildGroupbyCombinations(modifiedFormData);
+
+ expect(combinations).toEqual([
+ { rows: [], columns: [] },
+ { rows: ['row'], columns: [] },
+ ]);
+
+ expect(combinations).toHaveLength(2);
+});
+
+test('should handle single dimension in columns only', () => {
+ const modifiedFormData = {
+ ...baseFormData,
+ groupbyRows: [],
+ groupbyColumns: ['col'],
+ };
+
+ const combinations = buildGroupbyCombinations(modifiedFormData);
+
+ expect(combinations).toEqual([
+ { rows: [], columns: [] },
+ { rows: [], columns: ['col'] },
+ ]);
+
+ expect(combinations).toHaveLength(2);
+});
+
+test('should handle empty groupby arrays', () => {
+ const modifiedFormData = {
+ ...baseFormData,
+ groupbyRows: [],
+ groupbyColumns: [],
+ };
+
+ const combinations = buildGroupbyCombinations(modifiedFormData);
+
+ expect(combinations).toEqual([{ rows: [], columns: [] }]);
+
+ expect(combinations).toHaveLength(1);
+});
+
+test('should work with combineMetric and transposed pivot', () => {
+ const modifiedFormData = {
+ ...baseFormData,
+ transposePivot: true,
+ combineMetric: true,
+ metricsLayout: MetricsLayoutEnum.COLUMNS,
+ };
+
+ const combinations = buildGroupbyCombinations(modifiedFormData);
+
+ expect(combinations).toEqual([
+ { rows: [], columns: ['row1', 'row2'] },
+ { rows: ['col1'], columns: ['row1', 'row2'] },
+ { rows: ['col1', 'col2'], columns: ['row1', 'row2'] },
+ ]);
+});
+
+test('should handle combineMetric with empty arrays correctly', () => {
+ const modifiedFormData = {
+ ...baseFormData,
+ groupbyRows: [],
+ groupbyColumns: ['col'],
+ combineMetric: true,
+ metricsLayout: MetricsLayoutEnum.ROWS,
+ };
+
+ const combinations = buildGroupbyCombinations(modifiedFormData);
+
+ expect(combinations).toEqual([
+ { rows: [], columns: [] },
+ { rows: [], columns: ['col'] },
+ ]);
+});
+
+test('should work with large number of dimensions', () => {
+ const modifiedFormData = {
+ ...baseFormData,
+ groupbyRows: ['r1', 'r2', 'r3', 'r4'],
+ groupbyColumns: ['c1', 'c2', 'c3'],
+ };
+
+ const combinations = buildGroupbyCombinations(modifiedFormData);
+
+ expect(combinations).toHaveLength(20);
+
+ expect(combinations).toContainEqual({ rows: [], columns: [] });
+ expect(combinations).toContainEqual({
+ rows: ['r1', 'r2', 'r3', 'r4'],
+ columns: ['c1', 'c2', 'c3'],
+ });
+});
+
+test('isAdditiveMetric: SIMPLE metrics with additive aggregates are additive', () => {
+ expect(
+ isAdditiveMetric({
+ expressionType: 'SIMPLE',
+ aggregate: 'SUM',
+ column: { column_name: 'num' },
+ label: 'sum_num',
+ } as any),
+ ).toBe(true);
+ expect(
+ isAdditiveMetric({
+ expressionType: 'SIMPLE',
+ aggregate: 'COUNT',
+ column: { column_name: 'num' },
+ label: 'count_num',
+ } as any),
+ ).toBe(true);
+});
+
+test('isAdditiveMetric: non-additive aggregates, SQL, and saved metrics are not additive', () => {
+ expect(
+ isAdditiveMetric({
+ expressionType: 'SIMPLE',
+ aggregate: 'AVG',
+ column: { column_name: 'num' },
+ label: 'avg_num',
+ } as any),
+ ).toBe(false);
+ expect(
+ isAdditiveMetric({
+ expressionType: 'SIMPLE',
+ aggregate: 'COUNT_DISTINCT',
+ column: { column_name: 'name' },
+ label: 'distinct_names',
+ } as any),
+ ).toBe(false);
+ expect(
+ isAdditiveMetric({
+ expressionType: 'SQL',
+ sqlExpression: 'SUM(a) / SUM(b)',
+ label: 'ratio',
+ } as any),
+ ).toBe(false);
+ // saved-metric reference: aggregate unknown from form data -> non-additive
+ expect(isAdditiveMetric('count' as any)).toBe(false);
+});
+
+test('allMetricsAdditive: all additive vs any non-additive vs empty', () => {
+ const sum = {
+ expressionType: 'SIMPLE',
+ aggregate: 'SUM',
+ column: { column_name: 'a' },
+ label: 'a',
+ } as any;
Review Comment:
**Suggestion:** Define `sum` with an explicit metric interface/type instead of using `as any`. [custom_rule]
**Severity Level:** Minor ⚠️
<details>
<summary><b>Why it matters? 🤔 </b></summary>
The `sum` fixture is introduced in new TypeScript code and uses `as any`, which is a direct violation of the custom rule.
</details>
<details>
<summary><b>Rule source 📖 </b></summary>
.cursor/rules/dev-standard.mdc (line 16)
</details>
[](https://app.codeant.ai/fix-in-ide?tool=cursor&prompt_id=5bebaa441f404cb495bfb9ee7129c1e3&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset) [](https://app.codeant.ai/fix-in-ide?tool=vscode-claude&prompt_id=5bebaa441f404cb495bfb9ee7129c1e3&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset)
*(Use Cmd/Ctrl + Click for best experience)*
<details>
<summary><b>Prompt for AI Agent 🤖 </b></summary>
```mdx
This is a comment left during a code review.
**Path:** superset-frontend/plugins/plugin-chart-pivot-table/test/plugin/utilities.test.ts
**Line:** 289:295
**Comment:**
*Custom Rule: Define `sum` with an explicit metric interface/type instead of using `as any`.
Validate the correctness of the flagged issue. If correct, How can I resolve this? If you propose a fix, implement it and please make it concise.
Once fix is implemented, also check other comments on the same PR, and ask user if the user wants to fix the rest of the comments as well. if said yes, then fetch all the comments validate the correctness and implement a minimal fix
```
</details>
<a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F41184&comment_hash=73e679571912b09af091ebac7ae9e8b98579ad1b293cdce73a521989b1566aba&reaction=like'>👍</a> | <a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F41184&comment_hash=73e679571912b09af091ebac7ae9e8b98579ad1b293cdce73a521989b1566aba&reaction=dislike'>👎</a>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(table/pivot-table): correct non-additive totals/subtotals via DB rollup [POC for SIP-216] [superset]
Posted by "codeant-ai-for-open-source[bot] (via GitHub)" <gi...@apache.org>.
codeant-ai-for-open-source[bot] commented on code in PR #41184:
URL: https://github.com/apache/superset/pull/41184#discussion_r3564126627
##########
superset-frontend/plugins/plugin-chart-pivot-table/test/plugin/buildQuery.test.ts:
##########
@@ -56,9 +55,41 @@ const formData: PivotTableQueryFormData = {
currencyFormat: { symbol: 'USD', symbolPosition: 'prefix' },
};
-test('should build groupby with series in form data', () => {
+test('additive metrics use the fast-path: a single full-detail query', () => {
+ const { queries } = buildQuery({
+ ...formData,
+ metrics: [
+ {
+ expressionType: 'SIMPLE',
+ aggregate: 'SUM',
+ column: { column_name: 'num' },
+ label: 'sum_num',
+ },
+ ] as any,
+ });
+ expect(queries).toHaveLength(1);
+ // The single leaf query carries all dimensions (2 rows + 2 cols).
+ expect(queries[0].columns).toHaveLength(4);
+});
+
+test('non-additive metrics emit a single GROUPING SETS query with all levels', () => {
+ // 2 row dims x 2 col dims -> (2+1) x (2+1) = 9 rollup levels, carried as
+ // grouping_sets on a single query (saved-metric strings are non-additive).
const queryContext = buildQuery(formData);
+ expect(queryContext.queries).toHaveLength(1);
const [query] = queryContext.queries;
+ // The single query selects the full set of dimensions ...
+ expect(query.columns).toHaveLength(4);
+ // ... and requests every rollup level via grouping_sets (grand total = []).
+ expect((query as any).grouping_sets).toHaveLength(9);
+ expect((query as any).grouping_sets[0]).toEqual([]);
+ expect((query as any).grouping_sets[8]).toHaveLength(4);
Review Comment:
**Suggestion:** Introduce a typed query interface (including `grouping_sets`) and cast `query` to that concrete type instead of using `any` in assertions. [custom_rule]
**Severity Level:** Minor 🧹
<details>
<summary><b>Why it matters? ⭐ </b></summary>
The new assertions cast `query` to `any` three times, which is a direct TypeScript `any` usage in changed code. That matches the no-any-types rule violation exactly.
</details>
<details>
<summary><b>Rule source 📖 </b></summary>
.cursor/rules/dev-standard.mdc (line 16)
</details>
[](https://app.codeant.ai/fix-in-ide?tool=cursor&prompt_id=8f605b8e8a2b4cb2a9425cae9fc3fcb8&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset) [](https://app.codeant.ai/fix-in-ide?tool=vscode-claude&prompt_id=8f605b8e8a2b4cb2a9425cae9fc3fcb8&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset)
*(Use Cmd/Ctrl + Click for best experience)*
<details>
<summary><b>Prompt for AI Agent 🤖 </b></summary>
```mdx
This is a comment left during a code review.
**Path:** superset-frontend/plugins/plugin-chart-pivot-table/test/plugin/buildQuery.test.ts
**Line:** 75:86
**Comment:**
*Custom Rule: Introduce a typed query interface (including `grouping_sets`) and cast `query` to that concrete type instead of using `any` in assertions.
Validate the correctness of the flagged issue. If correct, How can I resolve this? If you propose a fix, implement it and please make it concise.
Once fix is implemented, also check other comments on the same PR, and ask user if the user wants to fix the rest of the comments as well. if said yes, then fetch all the comments validate the correctness and implement a minimal fix
```
</details>
<a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F41184&comment_hash=4eda982ff6cc90118d8e7ca4331819a044db7291dd30bc9a9e941cd24ceb7f27&reaction=like'>👍</a> | <a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F41184&comment_hash=4eda982ff6cc90118d8e7ca4331819a044db7291dd30bc9a9e941cd24ceb7f27&reaction=dislike'>👎</a>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(table/pivot-table): correct non-additive totals/subtotals via DB rollup [POC for SIP-216] [superset]
Posted by "rusackas (via GitHub)" <gi...@apache.org>.
rusackas commented on code in PR #41184:
URL: https://github.com/apache/superset/pull/41184#discussion_r3564072074
##########
superset/common/query_context_processor.py:
##########
@@ -252,9 +255,58 @@ def get_query_result(self, query_object: QueryObject) -> QueryResult:
This method delegates to the datasource's get_query_result method,
which handles query execution, normalization, time offsets, and
post-processing.
+
+ When the query requests rollup ``grouping_sets`` but the engine does not
+ support native ``GROUPING SETS``, fall back to one query per level and
+ concatenate the results with ``GROUPING()``-equivalent markers, so the
+ combined result matches the shape the native path produces (SIP.md,
+ phase 3b). Engines that support it run the single native query.
"""
+ if query_object.grouping_sets and not self._supports_grouping_sets():
+ return self._grouping_sets_fallback(query_object)
return self._qc_datasource.get_query_result(query_object)
+ def _supports_grouping_sets(self) -> bool:
+ engine_spec = getattr(self._qc_datasource, "db_engine_spec", None)
Review Comment:
Added explicit type annotation (`BaseEngineSpec | None`) for `engine_spec`.
##########
superset/common/query_context_processor.py:
##########
@@ -252,9 +255,58 @@ def get_query_result(self, query_object: QueryObject) -> QueryResult:
This method delegates to the datasource's get_query_result method,
which handles query execution, normalization, time offsets, and
post-processing.
+
+ When the query requests rollup ``grouping_sets`` but the engine does not
+ support native ``GROUPING SETS``, fall back to one query per level and
+ concatenate the results with ``GROUPING()``-equivalent markers, so the
+ combined result matches the shape the native path produces (SIP.md,
+ phase 3b). Engines that support it run the single native query.
"""
+ if query_object.grouping_sets and not self._supports_grouping_sets():
+ return self._grouping_sets_fallback(query_object)
return self._qc_datasource.get_query_result(query_object)
+ def _supports_grouping_sets(self) -> bool:
+ engine_spec = getattr(self._qc_datasource, "db_engine_spec", None)
+ return bool(engine_spec and engine_spec.supports_grouping_sets)
+
+ def _grouping_sets_fallback(self, query_object: QueryObject) -> QueryResult:
+ """
+ Emulate a GROUPING SETS query on engines without native support: run one
+ query per rollup level and concatenate, tagging each level's rows with
+ the same per-column markers the native path emits.
+ """
+ levels = query_object.grouping_sets
Review Comment:
Added `list[list[str]]` annotation for `levels`.
##########
superset/db_engine_specs/presto.py:
##########
@@ -166,6 +166,7 @@ class PrestoBaseEngineSpec(BaseEngineSpec, metaclass=ABCMeta):
supports_dynamic_schema = True
supports_catalog = supports_dynamic_catalog = supports_cross_catalog_queries = True
+ supports_grouping_sets = True
Review Comment:
Not adding this — none of the boolean capability flags on `BaseEngineSpec` (`supports_dynamic_schema`, `supports_catalog`, etc.) are type-annotated, so a one-off `: bool` here would be inconsistent with the class's existing convention rather than an improvement.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(table/pivot-table): correct non-additive totals/subtotals via DB rollup [POC for SIP-216] [superset]
Posted by "codeant-ai-for-open-source[bot] (via GitHub)" <gi...@apache.org>.
codeant-ai-for-open-source[bot] commented on code in PR #41184:
URL: https://github.com/apache/superset/pull/41184#discussion_r3555845735
##########
superset/common/grouping_sets.py:
##########
@@ -0,0 +1,116 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""
+SQL building blocks for the pivot-table non-additive totals optimization
+(SIP.md, phase 3b). When a datasource engine reports
+``supports_grouping_sets``, the N per-rollup-level queries can be collapsed into
+a single ``GROUPING SETS`` query: the database computes every level in one scan,
+and each returned row is attributed to its level via ``GROUPING()`` markers.
+
+These are the engine-agnostic SQL primitives. Wiring them into the query context
+(emitting one query and splitting the result back into per-level results) is the
+remaining integration; see SIP.md.
+"""
+
+from __future__ import annotations
+
+from collections.abc import Sequence
+
+import pandas as pd
+from sqlalchemy import func, tuple_
+from sqlalchemy.sql.elements import ColumnElement
+
+# Suffix for the per-column GROUPING() marker columns added to a GROUPING SETS
+# query. Chosen to be unlikely to collide with a real metric/column label.
+GROUPING_MARKER_SUFFIX = "__superset_grouping"
Review Comment:
**Suggestion:** Add an explicit type annotation to this module-level constant so it complies with the required type-hinting rule for annotatable variables. [custom_rule]
**Severity Level:** Minor ⚠️
<details>
<summary><b>Why it matters? 🤔 </b></summary>
This module-level constant is a simple string value that can be annotated, but it is declared without a type hint. That matches the rule requiring type hints on relevant variables that can be annotated.
</details>
<details>
<summary><b>Rule source 📖 </b></summary>
.cursor/rules/dev-standard.mdc (line 28)
</details>
[](https://app.codeant.ai/fix-in-ide?tool=cursor&prompt_id=b88ee1d9981948808b1e6f5347a0adb2&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset) [](https://app.codeant.ai/fix-in-ide?tool=vscode-claude&prompt_id=b88ee1d9981948808b1e6f5347a0adb2&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset)
*(Use Cmd/Ctrl + Click for best experience)*
<details>
<summary><b>Prompt for AI Agent 🤖 </b></summary>
```mdx
This is a comment left during a code review.
**Path:** superset/common/grouping_sets.py
**Line:** 40:40
**Comment:**
*Custom Rule: Add an explicit type annotation to this module-level constant so it complies with the required type-hinting rule for annotatable variables.
Validate the correctness of the flagged issue. If correct, How can I resolve this? If you propose a fix, implement it and please make it concise.
Once fix is implemented, also check other comments on the same PR, and ask user if the user wants to fix the rest of the comments as well. if said yes, then fetch all the comments validate the correctness and implement a minimal fix
```
</details>
<a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F41184&comment_hash=fd447a30ae1ac8fad98b2d8da8be51b48307da59579fe6d2a29222080c471bed&reaction=like'>👍</a> | <a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F41184&comment_hash=fd447a30ae1ac8fad98b2d8da8be51b48307da59579fe6d2a29222080c471bed&reaction=dislike'>👎</a>
##########
superset/db_engine_specs/base.py:
##########
@@ -560,6 +560,12 @@ class BaseEngineSpec: # pylint: disable=too-many-public-methods
# if True, database will be listed as option in the upload file form
supports_file_upload = True
+ # Whether the engine supports SQL GROUPING SETS / ROLLUP / CUBE. When True,
+ # consumers (e.g. the pivot table's non-additive totals) can collapse the
+ # per-rollup-level queries into a single GROUPING SETS query instead of
+ # issuing one query per level. Conservative default of False; engines opt in.
+ supports_grouping_sets = False
Review Comment:
**Suggestion:** Add an explicit boolean type annotation to this newly introduced class attribute to comply with the type-hint requirement. [custom_rule]
**Severity Level:** Minor ⚠️
<details>
<summary><b>Why it matters? 🤔 </b></summary>
The new class attribute is a boolean constant assigned without a type hint. The rule explicitly requires type hints on new or modified variables that can be annotated, so this is a real violation.
</details>
<details>
<summary><b>Rule source 📖 </b></summary>
.cursor/rules/dev-standard.mdc (line 28)
</details>
[](https://app.codeant.ai/fix-in-ide?tool=cursor&prompt_id=53bd1a7673a649918ea161db912faa06&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset) [](https://app.codeant.ai/fix-in-ide?tool=vscode-claude&prompt_id=53bd1a7673a649918ea161db912faa06&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset)
*(Use Cmd/Ctrl + Click for best experience)*
<details>
<summary><b>Prompt for AI Agent 🤖 </b></summary>
```mdx
This is a comment left during a code review.
**Path:** superset/db_engine_specs/base.py
**Line:** 567:567
**Comment:**
*Custom Rule: Add an explicit boolean type annotation to this newly introduced class attribute to comply with the type-hint requirement.
Validate the correctness of the flagged issue. If correct, How can I resolve this? If you propose a fix, implement it and please make it concise.
Once fix is implemented, also check other comments on the same PR, and ask user if the user wants to fix the rest of the comments as well. if said yes, then fetch all the comments validate the correctness and implement a minimal fix
```
</details>
<a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F41184&comment_hash=3b382d9496e03986a168eb41a18ad90e83ef6be104dbdac6d606dc556fcd7d35&reaction=like'>👍</a> | <a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F41184&comment_hash=3b382d9496e03986a168eb41a18ad90e83ef6be104dbdac6d606dc556fcd7d35&reaction=dislike'>👎</a>
##########
superset/common/query_object.py:
##########
@@ -157,6 +158,7 @@ def __init__( # pylint: disable=too-many-locals, too-many-arguments
self.series_limit = series_limit
self.series_limit_metric = series_limit_metric
self.group_others_when_limit_reached = group_others_when_limit_reached
+ self.grouping_sets = grouping_sets or []
Review Comment:
**Suggestion:** Add an explicit type annotation for the new instance attribute so `grouping_sets` is declared alongside other typed class attributes. [custom_rule]
**Severity Level:** Minor ⚠️
<details>
<summary><b>Why it matters? 🤔 </b></summary>
The new instance attribute `grouping_sets` is assigned without an explicit type annotation, while nearby class attributes are typed. This matches the rule for flagging newly introduced Python variables that can be annotated.
</details>
<details>
<summary><b>Rule source 📖 </b></summary>
.cursor/rules/dev-standard.mdc (line 28)
</details>
[](https://app.codeant.ai/fix-in-ide?tool=cursor&prompt_id=0e974df621bd4c2db3fa774d6e42b52a&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset) [](https://app.codeant.ai/fix-in-ide?tool=vscode-claude&prompt_id=0e974df621bd4c2db3fa774d6e42b52a&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset)
*(Use Cmd/Ctrl + Click for best experience)*
<details>
<summary><b>Prompt for AI Agent 🤖 </b></summary>
```mdx
This is a comment left during a code review.
**Path:** superset/common/query_object.py
**Line:** 161:161
**Comment:**
*Custom Rule: Add an explicit type annotation for the new instance attribute so `grouping_sets` is declared alongside other typed class attributes.
Validate the correctness of the flagged issue. If correct, How can I resolve this? If you propose a fix, implement it and please make it concise.
Once fix is implemented, also check other comments on the same PR, and ask user if the user wants to fix the rest of the comments as well. if said yes, then fetch all the comments validate the correctness and implement a minimal fix
```
</details>
<a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F41184&comment_hash=7aafc6b02a17730c0819bf78e1221615136c3f66ce83bdc239729578dc972d09&reaction=like'>👍</a> | <a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F41184&comment_hash=7aafc6b02a17730c0819bf78e1221615136c3f66ce83bdc239729578dc972d09&reaction=dislike'>👎</a>
##########
superset/db_engine_specs/bigquery.py:
##########
@@ -287,6 +287,7 @@ class BigQueryEngineSpec(BaseEngineSpec): # pylint: disable=too-many-public-met
supports_catalog = supports_dynamic_catalog = supports_cross_catalog_queries = True
supports_dynamic_schema = True
+ supports_grouping_sets = True
Review Comment:
**Suggestion:** Add an explicit boolean type annotation to this newly introduced class attribute. [custom_rule]
**Severity Level:** Minor ⚠️
<details>
<summary><b>Why it matters? 🤔 </b></summary>
This is a newly added class attribute in Python and its type is clearly inferable as bool, so it can be annotated as `supports_grouping_sets: bool = True` under the type-hint requirement.
</details>
<details>
<summary><b>Rule source 📖 </b></summary>
.cursor/rules/dev-standard.mdc (line 28)
</details>
[](https://app.codeant.ai/fix-in-ide?tool=cursor&prompt_id=dcca576b97b340f9a9cfe05c7912535d&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset) [](https://app.codeant.ai/fix-in-ide?tool=vscode-claude&prompt_id=dcca576b97b340f9a9cfe05c7912535d&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset)
*(Use Cmd/Ctrl + Click for best experience)*
<details>
<summary><b>Prompt for AI Agent 🤖 </b></summary>
```mdx
This is a comment left during a code review.
**Path:** superset/db_engine_specs/bigquery.py
**Line:** 290:290
**Comment:**
*Custom Rule: Add an explicit boolean type annotation to this newly introduced class attribute.
Validate the correctness of the flagged issue. If correct, How can I resolve this? If you propose a fix, implement it and please make it concise.
Once fix is implemented, also check other comments on the same PR, and ask user if the user wants to fix the rest of the comments as well. if said yes, then fetch all the comments validate the correctness and implement a minimal fix
```
</details>
<a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F41184&comment_hash=4d82b19e4f258106ea05aa390db0e051953f7823edc33d124603579238201a40&reaction=like'>👍</a> | <a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F41184&comment_hash=4d82b19e4f258106ea05aa390db0e051953f7823edc33d124603579238201a40&reaction=dislike'>👎</a>
##########
superset-frontend/plugins/plugin-chart-pivot-table/test/plugin/buildQuery.test.ts:
##########
@@ -56,9 +55,41 @@ const formData: PivotTableQueryFormData = {
currencyFormat: { symbol: 'USD', symbolPosition: 'prefix' },
};
-test('should build groupby with series in form data', () => {
+test('additive metrics use the fast-path: a single full-detail query', () => {
+ const { queries } = buildQuery({
+ ...formData,
+ metrics: [
+ {
+ expressionType: 'SIMPLE',
+ aggregate: 'SUM',
+ column: { column_name: 'num' },
+ label: 'sum_num',
+ },
+ ] as any,
+ });
+ expect(queries).toHaveLength(1);
+ // The single leaf query carries all dimensions (2 rows + 2 cols).
+ expect(queries[0].columns).toHaveLength(4);
+});
+
+test('non-additive metrics emit a single GROUPING SETS query with all levels', () => {
+ // 2 row dims x 2 col dims -> (2+1) x (2+1) = 9 rollup levels, carried as
+ // grouping_sets on a single query (saved-metric strings are non-additive).
const queryContext = buildQuery(formData);
+ expect(queryContext.queries).toHaveLength(1);
const [query] = queryContext.queries;
+ // The single query selects the full set of dimensions ...
+ expect(query.columns).toHaveLength(4);
+ // ... and requests every rollup level via grouping_sets (grand total = []).
+ expect((query as any).grouping_sets).toHaveLength(9);
Review Comment:
**Suggestion:** Avoid casting `query` to `any` and use a typed query interface that includes `grouping_sets`. [custom_rule]
**Severity Level:** Minor ⚠️
<details>
<summary><b>Why it matters? 🤔 </b></summary>
The changed test uses `query as any` to access `grouping_sets`, which is exactly the kind of `any` usage the rule forbids in new or modified TypeScript code.
</details>
<details>
<summary><b>Rule source 📖 </b></summary>
.cursor/rules/dev-standard.mdc (line 16)
</details>
[](https://app.codeant.ai/fix-in-ide?tool=cursor&prompt_id=5681f28a9f524fdea5b6b616353c9ae7&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset) [](https://app.codeant.ai/fix-in-ide?tool=vscode-claude&prompt_id=5681f28a9f524fdea5b6b616353c9ae7&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset)
*(Use Cmd/Ctrl + Click for best experience)*
<details>
<summary><b>Prompt for AI Agent 🤖 </b></summary>
```mdx
This is a comment left during a code review.
**Path:** superset-frontend/plugins/plugin-chart-pivot-table/test/plugin/buildQuery.test.ts
**Line:** 84:84
**Comment:**
*Custom Rule: Avoid casting `query` to `any` and use a typed query interface that includes `grouping_sets`.
Validate the correctness of the flagged issue. If correct, How can I resolve this? If you propose a fix, implement it and please make it concise.
Once fix is implemented, also check other comments on the same PR, and ask user if the user wants to fix the rest of the comments as well. if said yes, then fetch all the comments validate the correctness and implement a minimal fix
```
</details>
<a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F41184&comment_hash=f98638557a803b6d4c59578ba3a0e79bd8840b29dc2e8cc05899a474ffaf883c&reaction=like'>👍</a> | <a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F41184&comment_hash=f98638557a803b6d4c59578ba3a0e79bd8840b29dc2e8cc05899a474ffaf883c&reaction=dislike'>👎</a>
##########
superset-frontend/plugins/plugin-chart-pivot-table/test/plugin/buildQuery.test.ts:
##########
@@ -56,9 +55,41 @@ const formData: PivotTableQueryFormData = {
currencyFormat: { symbol: 'USD', symbolPosition: 'prefix' },
};
-test('should build groupby with series in form data', () => {
+test('additive metrics use the fast-path: a single full-detail query', () => {
+ const { queries } = buildQuery({
+ ...formData,
+ metrics: [
+ {
+ expressionType: 'SIMPLE',
+ aggregate: 'SUM',
+ column: { column_name: 'num' },
+ label: 'sum_num',
+ },
+ ] as any,
Review Comment:
**Suggestion:** Replace this `any` cast with a concrete metric array type so the test data is typed explicitly. [custom_rule]
**Severity Level:** Minor ⚠️
<details>
<summary><b>Why it matters? 🤔 </b></summary>
The new test code explicitly casts the metric object array to `any`, which matches the rule forbidding `any` in changed TypeScript code. A concrete metric type should be used instead.
</details>
<details>
<summary><b>Rule source 📖 </b></summary>
.cursor/rules/dev-standard.mdc (line 16)
</details>
[](https://app.codeant.ai/fix-in-ide?tool=cursor&prompt_id=e35f122c33c849bf903e929c74f22693&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset) [](https://app.codeant.ai/fix-in-ide?tool=vscode-claude&prompt_id=e35f122c33c849bf903e929c74f22693&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset)
*(Use Cmd/Ctrl + Click for best experience)*
<details>
<summary><b>Prompt for AI Agent 🤖 </b></summary>
```mdx
This is a comment left during a code review.
**Path:** superset-frontend/plugins/plugin-chart-pivot-table/test/plugin/buildQuery.test.ts
**Line:** 68:68
**Comment:**
*Custom Rule: Replace this `any` cast with a concrete metric array type so the test data is typed explicitly.
Validate the correctness of the flagged issue. If correct, How can I resolve this? If you propose a fix, implement it and please make it concise.
Once fix is implemented, also check other comments on the same PR, and ask user if the user wants to fix the rest of the comments as well. if said yes, then fetch all the comments validate the correctness and implement a minimal fix
```
</details>
<a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F41184&comment_hash=2a083ee7fede73106bfaf8a0e06ba5d480170176a7ad34d95b9f2d72909f9ebd&reaction=like'>👍</a> | <a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F41184&comment_hash=2a083ee7fede73106bfaf8a0e06ba5d480170176a7ad34d95b9f2d72909f9ebd&reaction=dislike'>👎</a>
##########
superset-frontend/plugins/plugin-chart-pivot-table/test/plugin/buildQuery.test.ts:
##########
@@ -56,9 +55,41 @@ const formData: PivotTableQueryFormData = {
currencyFormat: { symbol: 'USD', symbolPosition: 'prefix' },
};
-test('should build groupby with series in form data', () => {
+test('additive metrics use the fast-path: a single full-detail query', () => {
+ const { queries } = buildQuery({
+ ...formData,
+ metrics: [
+ {
+ expressionType: 'SIMPLE',
+ aggregate: 'SUM',
+ column: { column_name: 'num' },
+ label: 'sum_num',
+ },
+ ] as any,
+ });
+ expect(queries).toHaveLength(1);
+ // The single leaf query carries all dimensions (2 rows + 2 cols).
+ expect(queries[0].columns).toHaveLength(4);
+});
+
+test('non-additive metrics emit a single GROUPING SETS query with all levels', () => {
+ // 2 row dims x 2 col dims -> (2+1) x (2+1) = 9 rollup levels, carried as
+ // grouping_sets on a single query (saved-metric strings are non-additive).
const queryContext = buildQuery(formData);
+ expect(queryContext.queries).toHaveLength(1);
const [query] = queryContext.queries;
+ // The single query selects the full set of dimensions ...
+ expect(query.columns).toHaveLength(4);
+ // ... and requests every rollup level via grouping_sets (grand total = []).
+ expect((query as any).grouping_sets).toHaveLength(9);
+ expect((query as any).grouping_sets[0]).toEqual([]);
Review Comment:
**Suggestion:** Replace the `any` cast with a concrete typed structure before checking individual grouping set entries. [custom_rule]
**Severity Level:** Minor ⚠️
<details>
<summary><b>Why it matters? 🤔 </b></summary>
This is another direct `any` cast introduced in the changed TypeScript test, so it violates the no-`any` rule.
</details>
<details>
<summary><b>Rule source 📖 </b></summary>
.cursor/rules/dev-standard.mdc (line 16)
</details>
[](https://app.codeant.ai/fix-in-ide?tool=cursor&prompt_id=e457f48d5f2e4b59a585939378217210&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset) [](https://app.codeant.ai/fix-in-ide?tool=vscode-claude&prompt_id=e457f48d5f2e4b59a585939378217210&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset)
*(Use Cmd/Ctrl + Click for best experience)*
<details>
<summary><b>Prompt for AI Agent 🤖 </b></summary>
```mdx
This is a comment left during a code review.
**Path:** superset-frontend/plugins/plugin-chart-pivot-table/test/plugin/buildQuery.test.ts
**Line:** 85:85
**Comment:**
*Custom Rule: Replace the `any` cast with a concrete typed structure before checking individual grouping set entries.
Validate the correctness of the flagged issue. If correct, How can I resolve this? If you propose a fix, implement it and please make it concise.
Once fix is implemented, also check other comments on the same PR, and ask user if the user wants to fix the rest of the comments as well. if said yes, then fetch all the comments validate the correctness and implement a minimal fix
```
</details>
<a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F41184&comment_hash=1c87bc670ed2dbedbf8bb31d7d9711b69cba6f2db1ecfdf24019429e56d27342&reaction=like'>👍</a> | <a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F41184&comment_hash=1c87bc670ed2dbedbf8bb31d7d9711b69cba6f2db1ecfdf24019429e56d27342&reaction=dislike'>👎</a>
##########
superset-frontend/plugins/plugin-chart-pivot-table/test/react-pivottable/nonAdditiveTotals.test.ts:
##########
@@ -0,0 +1,69 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+/**
+ * Pivot-table acceptance cases for non-additive metric totals (see SIP.md).
+ *
+ * The pivot computes its grand total / subtotals client-side using the
+ * `aggregators` from react-pivottable, applied to the already-aggregated cell
+ * values. For a non-additive metric (a ratio like SUM(actual)/SUM(target))
+ * that is wrong: summing the per-group ratios is meaningless. This pins the
+ * mechanism behind #25747 / #32260 against the real production aggregator.
+ *
+ * The fix is not at this layer (no client-side aggregator over divided cells
+ * can recover the right answer) - it requires the DB-computed per-level rollup
+ * queries from phase 2. The `test.failing` case documents that target.
+ */
+import { aggregators } from '../../src/react-pivottable/utilities';
+
+// Per-group completion ratios (actual/target), as the single pivot query
+// returns them today.
+const ratioCells = [{ completion: 10 / 40 }, { completion: 30 / 60 }]; // 0.25, 0.5
+
+// Correct grand total: SUM(actual)/SUM(target) = 40/100 = 0.4
+const CORRECT_COMPLETION = 0.4;
+
+function sumOf(field: string, rows: Record<string, number>[]): number {
+ // Build the real "Sum" aggregator the pivot uses for totals.
+ const aggregator = (aggregators as Record<string, any>).Sum([field])(
Review Comment:
**Suggestion:** Replace the `any` cast with a concrete aggregator interface/type so the `Sum` access and return value are strongly typed. [custom_rule]
**Severity Level:** Minor ⚠️
<details>
<summary><b>Why it matters? 🤔 </b></summary>
The code uses `any` in a changed TypeScript file via `Record<string, any>`, which directly violates the no-any-types rule. A concrete type should be used instead.
</details>
<details>
<summary><b>Rule source 📖 </b></summary>
.cursor/rules/dev-standard.mdc (line 16)
</details>
[](https://app.codeant.ai/fix-in-ide?tool=cursor&prompt_id=a84b07f8227d410fbf735386c820f276&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset) [](https://app.codeant.ai/fix-in-ide?tool=vscode-claude&prompt_id=a84b07f8227d410fbf735386c820f276&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset)
*(Use Cmd/Ctrl + Click for best experience)*
<details>
<summary><b>Prompt for AI Agent 🤖 </b></summary>
```mdx
This is a comment left during a code review.
**Path:** superset-frontend/plugins/plugin-chart-pivot-table/test/react-pivottable/nonAdditiveTotals.test.ts
**Line:** 44:44
**Comment:**
*Custom Rule: Replace the `any` cast with a concrete aggregator interface/type so the `Sum` access and return value are strongly typed.
Validate the correctness of the flagged issue. If correct, How can I resolve this? If you propose a fix, implement it and please make it concise.
Once fix is implemented, also check other comments on the same PR, and ask user if the user wants to fix the rest of the comments as well. if said yes, then fetch all the comments validate the correctness and implement a minimal fix
```
</details>
<a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F41184&comment_hash=af978e726f2c35c6bd974a6e671a0a7c6a1997cd5323554584d96246c24c3206&reaction=like'>👍</a> | <a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F41184&comment_hash=af978e726f2c35c6bd974a6e671a0a7c6a1997cd5323554584d96246c24c3206&reaction=dislike'>👎</a>
##########
superset-frontend/plugins/plugin-chart-pivot-table/test/plugin/buildQuery.test.ts:
##########
@@ -56,9 +55,41 @@ const formData: PivotTableQueryFormData = {
currencyFormat: { symbol: 'USD', symbolPosition: 'prefix' },
};
-test('should build groupby with series in form data', () => {
+test('additive metrics use the fast-path: a single full-detail query', () => {
+ const { queries } = buildQuery({
+ ...formData,
+ metrics: [
+ {
+ expressionType: 'SIMPLE',
+ aggregate: 'SUM',
+ column: { column_name: 'num' },
+ label: 'sum_num',
+ },
+ ] as any,
+ });
+ expect(queries).toHaveLength(1);
+ // The single leaf query carries all dimensions (2 rows + 2 cols).
+ expect(queries[0].columns).toHaveLength(4);
+});
+
+test('non-additive metrics emit a single GROUPING SETS query with all levels', () => {
+ // 2 row dims x 2 col dims -> (2+1) x (2+1) = 9 rollup levels, carried as
+ // grouping_sets on a single query (saved-metric strings are non-additive).
const queryContext = buildQuery(formData);
+ expect(queryContext.queries).toHaveLength(1);
const [query] = queryContext.queries;
+ // The single query selects the full set of dimensions ...
+ expect(query.columns).toHaveLength(4);
+ // ... and requests every rollup level via grouping_sets (grand total = []).
+ expect((query as any).grouping_sets).toHaveLength(9);
+ expect((query as any).grouping_sets[0]).toEqual([]);
+ expect((query as any).grouping_sets[8]).toHaveLength(4);
Review Comment:
**Suggestion:** Remove this `any` cast by typing the query object with a specific type that exposes `grouping_sets` length safely. [custom_rule]
**Severity Level:** Minor ⚠️
<details>
<summary><b>Why it matters? 🤔 </b></summary>
The code added in this hunk uses `any` again to access `grouping_sets`, which is a real violation of the rule against `any` in changed TypeScript code.
</details>
<details>
<summary><b>Rule source 📖 </b></summary>
.cursor/rules/dev-standard.mdc (line 16)
</details>
[](https://app.codeant.ai/fix-in-ide?tool=cursor&prompt_id=091b7f14be6a4301b6f893fe48ede44d&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset) [](https://app.codeant.ai/fix-in-ide?tool=vscode-claude&prompt_id=091b7f14be6a4301b6f893fe48ede44d&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset)
*(Use Cmd/Ctrl + Click for best experience)*
<details>
<summary><b>Prompt for AI Agent 🤖 </b></summary>
```mdx
This is a comment left during a code review.
**Path:** superset-frontend/plugins/plugin-chart-pivot-table/test/plugin/buildQuery.test.ts
**Line:** 86:86
**Comment:**
*Custom Rule: Remove this `any` cast by typing the query object with a specific type that exposes `grouping_sets` length safely.
Validate the correctness of the flagged issue. If correct, How can I resolve this? If you propose a fix, implement it and please make it concise.
Once fix is implemented, also check other comments on the same PR, and ask user if the user wants to fix the rest of the comments as well. if said yes, then fetch all the comments validate the correctness and implement a minimal fix
```
</details>
<a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F41184&comment_hash=a1a906908101502d2f13fd521e5eff914653358d3505a28eb3e43397e0dfea9b&reaction=like'>👍</a> | <a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F41184&comment_hash=a1a906908101502d2f13fd521e5eff914653358d3505a28eb3e43397e0dfea9b&reaction=dislike'>👎</a>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(table/pivot-table): correct non-additive totals/subtotals via DB rollup [SIP-216] [superset]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #41184:
URL: https://github.com/apache/superset/pull/41184#issuecomment-5008627764
🎪 [Showtime](https://github.com/mistercrunch/superset-showtime) deployed environment on [GHA](https://github.com/apache/superset/actions/runs/29621220321) for [67b8a00](https://github.com/apache/superset/commit/67b8a00)
• **Environment:** http://44.249.137.165:8080 (admin/admin)
• **Lifetime:** 48h auto-cleanup
• **Updates:** New commits create fresh environments automatically
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(table/pivot-table): correct non-additive totals/subtotals via DB rollup [SIP-216] [superset]
Posted by "rusackas (via GitHub)" <gi...@apache.org>.
rusackas commented on PR #41184:
URL: https://github.com/apache/superset/pull/41184#issuecomment-5041521649
Thanks for digging in, @aminghadersohi. Good lead... though it turned out not to be the last-row logic at all. `splitGroupingSetsResult` attributes each row to a rollup level via the `GROUPING()` marker columns, and the old test fixtures feed rows with no markers, so every row got filtered out and the table rendered empty. Fixed by treating marker-less results as leaf-level data (blank totals cells beat a blank chart), with a unit test to pin it down, and merged current `master` while I was in there. CI's green again.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(table/pivot-table): correct non-additive totals/subtotals via DB rollup [POC for SIP-216] [superset]
Posted by "codeant-ai-for-open-source[bot] (via GitHub)" <gi...@apache.org>.
codeant-ai-for-open-source[bot] commented on code in PR #41184:
URL: https://github.com/apache/superset/pull/41184#discussion_r3555863956
##########
superset-frontend/plugins/plugin-chart-pivot-table/src/PivotTableChart.tsx:
##########
@@ -341,22 +296,43 @@ export default function PivotTableChart(props: PivotTableProps) {
const unpivotedData = useMemo(
() =>
- data.reduce(
- (acc: Record<string, any>[], record: Record<string, any>) => [
- ...acc,
- ...metricNames
+ // `data` is now one entry per rollup level. Tag every record with the
+ // row/column dimension labels of the level that produced it (mirroring
+ // the METRIC_KEY injection used for the full rows/cols below) so
+ // PivotData can slot each pre-computed value without re-aggregating.
+ // buildGroupbyCombinations already applied transposePivot, so the level's
+ // groupby is display-oriented and is not transposed again here.
+ data.flatMap((query: QueryData) => {
+ let levelRows = query.groupby.rows.map(getColumnLabel);
+ let levelCols = query.groupby.columns.map(getColumnLabel);
+ if (metricsLayout === MetricsLayoutEnum.ROWS) {
+ levelRows = combineMetric
+ ? [...levelRows, METRIC_KEY]
+ : [METRIC_KEY, ...levelRows];
+ } else {
+ levelCols = combineMetric
+ ? [...levelCols, METRIC_KEY]
+ : [METRIC_KEY, ...levelCols];
+ }
+ return query.data.flatMap((record: Record<string, any>) =>
+ metricNames
.map((name: string) => ({
...record,
[METRIC_KEY]: name,
value: record[name],
// Mark currency column for per-cell currency detection in aggregators
__currencyColumn: currencyCodeColumn,
+ // The level this record belongs to (used by PivotData placement).
+ rows: levelRows,
+ columns: levelCols,
+ // Identify the metric pseudo-dimension so PivotData can feed the
+ // metric-collapsed totals (the opposite "Total" axis + corner).
+ __metricKey: METRIC_KEY,
}))
- .filter(record => record.value !== null),
- ],
- [],
- ),
- [data, metricNames, currencyCodeColumn],
+ .filter(r => r.value !== null),
Review Comment:
**Suggestion:** Filtering out every metric row where the value is null removes those dimension keys from PivotData entirely, so rows/columns that legitimately have null metrics disappear instead of rendering as empty cells. Keep the record and let the renderer show a blank/null value so table structure remains correct. [logic error]
<details>
<summary><b>Severity Level:</b> Major ⚠️</summary>
```mdx
- ❌ Pivot table omits rows with only null metric values.
- ⚠️ Column headers misalign when null rows are pruned.
```
</details>
<details>
<summary><b>Steps of Reproduction ✅ </b></summary>
```mdx
1. Configure a Pivot Table chart in the UI with a metric that can legitimately return NULL
at some rollup level (e.g., an AVG over empty groups) and run the query, producing
`queriesData[0].data` with null metric values, which is passed into `transformProps()` at
`superset-frontend/plugins/plugin-chart-pivot-table/src/plugin/transformProps.ts:54`.
2. Inside `transformProps`, the `data: QueryData[]` structure is built (additive or
grouping-sets path) at `transformProps.ts:105-144`, preserving rows where certain metric
fields are `null` and associating each with its `groupby` level.
3. `PivotTableChart` is rendered with those `data` props at
`superset-frontend/plugins/plugin-chart-pivot-table/src/PivotTableChart.tsx:181-215`, and
computes `unpivotedData` in the `useMemo` block at `PivotTableChart.tsx:297-335` by
mapping each `record` and `metricNames` entry into `{ ...record, [METRIC_KEY]: name,
value: record[name], rows, columns, __metricKey }`.
4. The final step in that `useMemo` applies `.filter(r => r.value !== null)` at
`PivotTableChart.tsx:332`, dropping every record whose metric `value` is `null`; when all
metrics for a given dimension combination are null at a rollup level, no records remain
for that row/column in `unpivotedData`, so the `<PivotTable>` rendered at
`PivotTableChart.tsx:671-685` never sees that dimension key and the corresponding
row/column disappears instead of showing an empty cell.
```
</details>
[](https://app.codeant.ai/fix-in-ide?tool=cursor&prompt_id=25c7b0eaa9fc4fc2aa94a467c8208099&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset) [](https://app.codeant.ai/fix-in-ide?tool=vscode-claude&prompt_id=25c7b0eaa9fc4fc2aa94a467c8208099&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset)
*(Use Cmd/Ctrl + Click for best experience)*
<details>
<summary><b>Prompt for AI Agent 🤖 </b></summary>
```mdx
This is a comment left during a code review.
**Path:** superset-frontend/plugins/plugin-chart-pivot-table/src/PivotTableChart.tsx
**Line:** 332:332
**Comment:**
*Logic Error: Filtering out every metric row where the value is null removes those dimension keys from PivotData entirely, so rows/columns that legitimately have null metrics disappear instead of rendering as empty cells. Keep the record and let the renderer show a blank/null value so table structure remains correct.
Validate the correctness of the flagged issue. If correct, How can I resolve this? If you propose a fix, implement it and please make it concise.
Once fix is implemented, also check other comments on the same PR, and ask user if the user wants to fix the rest of the comments as well. if said yes, then fetch all the comments validate the correctness and implement a minimal fix
```
</details>
<a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F41184&comment_hash=a1621286e7161a3e15f5c24652ccc356b8029fba630f7d04678cc2b92a5f0e33&reaction=like'>👍</a> | <a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F41184&comment_hash=a1621286e7161a3e15f5c24652ccc356b8029fba630f7d04678cc2b92a5f0e33&reaction=dislike'>👎</a>
##########
superset-frontend/plugins/plugin-chart-pivot-table/test/plugin/buildQuery.test.ts:
##########
@@ -56,9 +55,41 @@ const formData: PivotTableQueryFormData = {
currencyFormat: { symbol: 'USD', symbolPosition: 'prefix' },
};
-test('should build groupby with series in form data', () => {
+test('additive metrics use the fast-path: a single full-detail query', () => {
+ const { queries } = buildQuery({
+ ...formData,
+ metrics: [
+ {
+ expressionType: 'SIMPLE',
+ aggregate: 'SUM',
+ column: { column_name: 'num' },
+ label: 'sum_num',
+ },
+ ] as any,
+ });
+ expect(queries).toHaveLength(1);
+ // The single leaf query carries all dimensions (2 rows + 2 cols).
+ expect(queries[0].columns).toHaveLength(4);
+});
+
+test('non-additive metrics emit a single GROUPING SETS query with all levels', () => {
+ // 2 row dims x 2 col dims -> (2+1) x (2+1) = 9 rollup levels, carried as
+ // grouping_sets on a single query (saved-metric strings are non-additive).
const queryContext = buildQuery(formData);
+ expect(queryContext.queries).toHaveLength(1);
const [query] = queryContext.queries;
+ // The single query selects the full set of dimensions ...
+ expect(query.columns).toHaveLength(4);
+ // ... and requests every rollup level via grouping_sets (grand total = []).
+ expect((query as any).grouping_sets).toHaveLength(9);
+ expect((query as any).grouping_sets[0]).toEqual([]);
+ expect((query as any).grouping_sets[8]).toHaveLength(4);
+});
+
+test('should build groupby with series in form data', () => {
+ const queryContext = buildQuery(formData);
+ // Multi-query rollup: the full-detail level is the last query (grand total is first).
+ const query = queryContext.queries[queryContext.queries.length - 1];
Review Comment:
**Suggestion:** The inline comment says this path is a “multi-query rollup,” but the new behavior in this file asserts a single-query approach with grouping sets; this contradiction makes the test misleading and can cause future regressions to be misinterpreted. Update the comment to match the actual single-query contract. [comment mismatch]
<details>
<summary><b>Severity Level:</b> Minor 🧹</summary>
```mdx
- ⚠️ Pivot table buildQuery tests misdocument single-query behavior.
- ⚠️ Future refactors may target non-existent multi-query rollup.
```
</details>
<details>
<summary><b>Steps of Reproduction ✅ </b></summary>
```mdx
1. Open
`superset-frontend/plugins/plugin-chart-pivot-table/test/plugin/buildQuery.test.ts` and
locate the test `non-additive metrics emit a single GROUPING SETS query with all levels`
at lines 75-87, which asserts `queryContext.queries` has length 1, i.e. the pivot-table
buildQuery path produces a single query for non-additive metrics.
2. In the same file, locate the subsequent test `should build groupby with series in form
data` at lines 89-105; inside it, lines 91-92 contain the comment `// Multi-query rollup:
the full-detail level is the last query (grand total is first).` immediately before `const
query = queryContext.queries[queryContext.queries.length - 1];`, implying multiple queries
are expected.
3. Open `superset-frontend/plugins/plugin-chart-pivot-table/src/plugin/buildQuery.ts` and
examine `buildQuery` at lines 64-112: the function always calls
`buildQueryContext(formData, ...)` with a callback that returns a single-element array `[
{ ...baseQueryObject, columns: fullColumns, ...(groupingSets ? { grouping_sets:
groupingSets } : {}) } ]`, proving that the frontend now always emits exactly one query
object, regardless of additive vs non-additive metrics.
4. Open `superset/common/query_context_processor.py` and inspect `get_query_result` at
lines 19-27: even when `query_object.grouping_sets` is set and `_supports_grouping_sets()`
is false, the multi-query fallback `_grouping_sets_fallback` at lines 33-64 is internal to
the processor; callers (including chart data and tests) still operate on a single
`QueryObject` in `QueryContext.queries`, confirming the test comment’s “Multi-query
rollup” wording is outdated and misleading relative to the current single-query contract.
```
</details>
[](https://app.codeant.ai/fix-in-ide?tool=cursor&prompt_id=00337b91326b4822b37d537c7ddad02b&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset) [](https://app.codeant.ai/fix-in-ide?tool=vscode-claude&prompt_id=00337b91326b4822b37d537c7ddad02b&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset)
*(Use Cmd/Ctrl + Click for best experience)*
<details>
<summary><b>Prompt for AI Agent 🤖 </b></summary>
```mdx
This is a comment left during a code review.
**Path:** superset-frontend/plugins/plugin-chart-pivot-table/test/plugin/buildQuery.test.ts
**Line:** 91:92
**Comment:**
*Comment Mismatch: The inline comment says this path is a “multi-query rollup,” but the new behavior in this file asserts a single-query approach with grouping sets; this contradiction makes the test misleading and can cause future regressions to be misinterpreted. Update the comment to match the actual single-query contract.
Validate the correctness of the flagged issue. If correct, How can I resolve this? If you propose a fix, implement it and please make it concise.
Once fix is implemented, also check other comments on the same PR, and ask user if the user wants to fix the rest of the comments as well. if said yes, then fetch all the comments validate the correctness and implement a minimal fix
```
</details>
<a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F41184&comment_hash=f15f963d14efd8f8587367812795cc0cebcec815da71f3548ee4713ce572b402&reaction=like'>👍</a> | <a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F41184&comment_hash=f15f963d14efd8f8587367812795cc0cebcec815da71f3548ee4713ce572b402&reaction=dislike'>👎</a>
##########
superset/charts/schemas.py:
##########
@@ -1358,6 +1358,17 @@ class Meta: # pylint: disable=too-few-public-methods
load_default=False,
allow_none=True,
)
+ grouping_sets = fields.List(
+ fields.List(fields.String()),
+ metadata={
+ "description": "Rollup levels for non-additive totals: each entry is "
+ "the list of groupby columns to group at that level (e.g. the empty "
+ "list is the grand total). When set and the engine supports it, the "
+ "levels are computed in a single GROUPING SETS query.",
+ },
+ load_default=None,
+ allow_none=True,
+ )
Review Comment:
**Suggestion:** The new `grouping_sets` field is fully user-controlled and has no size/shape validation. On engines without native `GROUPING SETS`, the backend executes one query per requested level, so a large payload can amplify into many DB queries and exhaust resources. Add validation to cap the number of levels and ensure each level is a subset of declared groupby columns. [performance]
<details>
<summary><b>Severity Level:</b> Critical 🚨</summary>
```mdx
- ❌ Chart data API can run unbounded per-level DB queries.
- ⚠️ Slow pivot or table charts on unsupported GROUPING SETS engines.
- ⚠️ Potential database resource exhaustion via oversized grouping_sets payloads.
```
</details>
<details>
<summary><b>Steps of Reproduction ✅ </b></summary>
```mdx
1. Open `superset/charts/schemas.py` and inspect `ChartDataQueryObjectSchema` starting at
line 1 (offset 1227); at lines 135-145 the new field `grouping_sets =
fields.List(fields.List(fields.String()), load_default=None, allow_none=True)` is defined
with only type information and description, but no `validate` constraints on list length,
element count, or allowed values.
2. Observe that `ChartDataQueryContextSchema` at lines 6-15 (offset 1498) has a `queries =
fields.List(fields.Nested(ChartDataQueryObjectSchema))` field and a `@post_load` method
`make_query_context` at lines 29-33 that constructs a `QueryContext`; this schema is used
wherever chart-style query contexts are created, including in
`superset/mcp_service/chart/tool/get_chart_data.py` at lines 5-7 and 612, and in the
public `/api/v1/chart/data` endpoint documented in `docs/developer_docs/api.mdx` lines
115-116.
3. Open `superset/common/query_object.py` and inspect `QueryObject.__init__` at lines
17-42 (offset 120): it accepts `grouping_sets: list[list[str]] | None = None` and stores
it as `self.grouping_sets = grouping_sets or []`, meaning any user-supplied
`grouping_sets` list from the schema becomes a concrete list of levels on the query object
with no additional validation.
4. Open `superset/common/query_context_processor.py` and examine `get_query_result` at
lines 19-27 and `_grouping_sets_fallback` at lines 33-64: when
`query_object.grouping_sets` is non-empty and `_supports_grouping_sets()` (lines 29-31)
returns false for the datasource’s engine spec, the processor runs
`_grouping_sets_fallback`, which does `levels = query_object.grouping_sets` and then loops
`for level in levels:` issuing `self._qc_datasource.get_query_result(sub_query)` once per
level. Because `grouping_sets` is fully user-controlled via `ChartDataQueryObjectSchema`
with no size limits, a client can send a `/api/v1/chart/data` request whose
`queries[0].grouping_sets` contains hundreds of levels, causing the fallback path on
engines without native GROUPING SETS support to execute hundreds of backend SQL queries
for a single chart data request, amplifying load without any server-side cap.
```
</details>
[](https://app.codeant.ai/fix-in-ide?tool=cursor&prompt_id=45a425ef9115430c9aef055bde281392&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset) [](https://app.codeant.ai/fix-in-ide?tool=vscode-claude&prompt_id=45a425ef9115430c9aef055bde281392&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset)
*(Use Cmd/Ctrl + Click for best experience)*
<details>
<summary><b>Prompt for AI Agent 🤖 </b></summary>
```mdx
This is a comment left during a code review.
**Path:** superset/charts/schemas.py
**Line:** 1361:1371
**Comment:**
*Performance: The new `grouping_sets` field is fully user-controlled and has no size/shape validation. On engines without native `GROUPING SETS`, the backend executes one query per requested level, so a large payload can amplify into many DB queries and exhaust resources. Add validation to cap the number of levels and ensure each level is a subset of declared groupby columns.
Validate the correctness of the flagged issue. If correct, How can I resolve this? If you propose a fix, implement it and please make it concise.
Once fix is implemented, also check other comments on the same PR, and ask user if the user wants to fix the rest of the comments as well. if said yes, then fetch all the comments validate the correctness and implement a minimal fix
```
</details>
<a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F41184&comment_hash=14dc8dae4a21eef37cc698c193df7f78cf6376e79fb6cb8deefa23fec520b930&reaction=like'>👍</a> | <a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F41184&comment_hash=14dc8dae4a21eef37cc698c193df7f78cf6376e79fb6cb8deefa23fec520b930&reaction=dislike'>👎</a>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(table/pivot-table): correct non-additive totals/subtotals via DB rollup [POC for SIP-216] [superset]
Posted by "codeant-ai-for-open-source[bot] (via GitHub)" <gi...@apache.org>.
codeant-ai-for-open-source[bot] commented on code in PR #41184:
URL: https://github.com/apache/superset/pull/41184#discussion_r3555865333
##########
superset-frontend/plugins/plugin-chart-pivot-table/src/react-pivottable/utilities.ts:
##########
@@ -1088,76 +1115,106 @@ class PivotData {
}
processRecord(record: PivotRecord): void {
- // this code is called in a tight loop
+ // this code is called in a tight loop.
+ // Each record is tagged (in PivotTableChart) with `rows`/`columns`: the
+ // dimension labels of the rollup level that produced it. The database has
+ // already aggregated that level, so we place the value into exactly one
+ // slot and store it verbatim (no re-aggregation). A key shorter than the
+ // full dimension list identifies a subtotal level. Records without tags
+ // (e.g. direct unit-test construction) fall back to the full dimension
+ // lists, i.e. they are treated as leaf rows.
+ const recordRows = (record.rows as unknown as string[]) ?? null;
+ const recordColumns = (record.columns as unknown as string[]) ?? null;
+ const levelRows = recordRows ?? (this.props.rows as string[]);
+ const levelColumns = recordColumns ?? (this.props.cols as string[]);
+
const colKey: string[] = [];
const rowKey: string[] = [];
- (this.props.cols as string[]).forEach((col: string) => {
+ levelColumns.forEach((col: string) => {
colKey.push(col in record ? String(record[col]) : 'null');
});
- (this.props.rows as string[]).forEach((row: string) => {
+ levelRows.forEach((row: string) => {
rowKey.push(row in record ? String(record[row]) : 'null');
});
- this.allTotal.push(record);
-
- const rowStart = this.subtotals.rowEnabled ? 1 : Math.max(1, rowKey.length);
- const colStart = this.subtotals.colEnabled ? 1 : Math.max(1, colKey.length);
-
- let isRowSubtotal;
- let isColSubtotal;
- for (let ri = rowStart; ri <= rowKey.length; ri += 1) {
- isRowSubtotal = ri < rowKey.length;
- const fRowKey = rowKey.slice(0, ri);
- const flatRowKey = flatKey(fRowKey);
- if (!this.rowTotals[flatRowKey]) {
- this.rowKeys.push(fRowKey);
- this.rowTotals[flatRowKey] = this.getFormattedAggregator(
- record,
- rowKey,
- )(this, fRowKey, []);
+ const flatRowKey = flatKey(rowKey);
+ const flatColKey = flatKey(colKey);
+
+ const isColSubtotal = colKey.length < (this.props.cols as string[]).length;
+ const isRowSubtotal = rowKey.length < (this.props.rows as string[]).length;
+
+ // Register/create the column-total slot the first time this colKey is seen.
+ if (colKey.length > 0 && !this.colTotals[flatColKey]) {
+ if (!isColSubtotal || this.subtotals.colEnabled) {
+ this.colKeys.push(colKey);
}
- this.rowTotals[flatRowKey].push(record);
- this.rowTotals[flatRowKey].isSubtotal = isRowSubtotal;
+ this.colTotals[flatColKey] = this.getFormattedAggregator(record, colKey)(
+ this,
+ [],
+ colKey,
+ );
}
-
- for (let ci = colStart; ci <= colKey.length; ci += 1) {
- isColSubtotal = ci < colKey.length;
- const fColKey = colKey.slice(0, ci);
- const flatColKey = flatKey(fColKey);
- if (!this.colTotals[flatColKey]) {
- this.colKeys.push(fColKey);
- this.colTotals[flatColKey] = this.getFormattedAggregator(
- record,
+ // Register/create the row-total slot the first time this rowKey is seen.
+ if (rowKey.length > 0 && !this.rowTotals[flatRowKey]) {
+ if (!isRowSubtotal || this.subtotals.rowEnabled) {
+ this.rowKeys.push(rowKey);
+ }
+ this.rowTotals[flatRowKey] = this.getFormattedAggregator(record, rowKey)(
+ this,
+ rowKey,
+ [],
+ );
+ }
+ // Create the body-cell slot.
+ if (rowKey.length > 0 && colKey.length > 0) {
+ if (!this.tree[flatRowKey]) {
+ this.tree[flatRowKey] = {};
+ }
+ if (!this.tree[flatRowKey][flatColKey]) {
+ this.tree[flatRowKey][flatColKey] = this.getFormattedAggregator(record)(
+ this,
+ rowKey,
colKey,
- )(this, [], fColKey);
+ );
}
+ }
+
+ // Place the value in exactly one slot, determined by the level.
+ if (rowKey.length === 0 && colKey.length === 0) {
+ this.allTotal.push(record);
+ } else if (rowKey.length === 0) {
this.colTotals[flatColKey].push(record);
this.colTotals[flatColKey].isSubtotal = isColSubtotal;
+ } else if (colKey.length === 0) {
+ this.rowTotals[flatRowKey].push(record);
+ this.rowTotals[flatRowKey].isSubtotal = isRowSubtotal;
+ } else {
+ this.tree[flatRowKey][flatColKey].push(record);
+ this.tree[flatRowKey][flatColKey].isRowSubtotal = isRowSubtotal;
+ this.tree[flatRowKey][flatColKey].isColSubtotal = isColSubtotal;
+ this.tree[flatRowKey][flatColKey].isSubtotal =
+ isRowSubtotal || isColSubtotal;
}
- // And now fill in for all the sub-cells.
- for (let ri = rowStart; ri <= rowKey.length; ri += 1) {
- isRowSubtotal = ri < rowKey.length;
- const fRowKey = rowKey.slice(0, ri);
- const flatRowKey = flatKey(fRowKey);
- if (!this.tree[flatRowKey]) {
- this.tree[flatRowKey] = {};
+ // Metric-collapse totals. The metric is a pseudo-dimension always present on
+ // one axis, so no rollup level produces an empty key on that axis -- which
+ // would leave the opposite "Total" axis and the grand-total corner empty.
+ // When a record's axis holds only the metric (no real dims there), its value
+ // is also the collapsed total for that axis, so mirror it into rowTotals /
+ // colTotals / allTotal. (For a single metric this equals the metric column;
+ // for multiple metrics it is the last metric -- a cross-metric total is not
+ // well defined and is left as future work.)
+ const metricKey = record.__metricKey as unknown as string | undefined;
+ if (metricKey) {
+ const realColCount = levelColumns.filter(c => c !== metricKey).length;
+ const realRowCount = levelRows.filter(r => r !== metricKey).length;
+ if (levelColumns.includes(metricKey) && realColCount === 0) {
+ if (rowKey.length === 0) this.allTotal.push(record);
+ else this.rowTotals[flatRowKey]?.push(record);
}
- for (let ci = colStart; ci <= colKey.length; ci += 1) {
- isColSubtotal = ci < colKey.length;
- const fColKey = colKey.slice(0, ci);
- const flatColKey = flatKey(fColKey);
- if (!this.tree[flatRowKey][flatColKey]) {
- this.tree[flatRowKey][flatColKey] = this.getFormattedAggregator(
- record,
- )(this, fRowKey, fColKey);
- }
- this.tree[flatRowKey][flatColKey].push(record);
-
- this.tree[flatRowKey][flatColKey].isRowSubtotal = isRowSubtotal;
- this.tree[flatRowKey][flatColKey].isColSubtotal = isColSubtotal;
- this.tree[flatRowKey][flatColKey].isSubtotal =
- isRowSubtotal || isColSubtotal;
+ if (levelRows.includes(metricKey) && realRowCount === 0) {
+ if (colKey.length === 0) this.allTotal.push(record);
+ else this.colTotals[flatColKey]?.push(record);
}
Review Comment:
**Suggestion:** The metric-collapse path pushes multiple metric records into the same total aggregators, but the new passthrough aggregator stores only one value per cell, so totals for multi-metric pivots are overwritten by whichever metric is processed last. This produces incorrect row/column/grand totals whenever metrics are collapsed on one axis. You need a deterministic aggregation strategy for these totals (or explicitly suppress/disable them) instead of repeatedly overwriting the same slot. [logic error]
<details>
<summary><b>Severity Level:</b> Major ⚠️</summary>
```mdx
❌ Multi-metric pivot totals display only last metric value.
⚠️ Users misinterpret collapsed totals on pivot table chart.
```
</details>
<details>
<summary><b>Steps of Reproduction ✅ </b></summary>
```mdx
1. Configure a Pivot Table chart using `PivotTableChart` (PivotTableChart.tsx lines
181–215) with at least two metrics in `metrics`, a layout that places the metric
pseudo-dimension (`METRIC_KEY`) on one axis (`metricsLayout` and `combineMetric`), and
enable row/column totals (`rowTotals`/`colTotals` true).
2. The backend returns `QueryData` objects; `unpivotedData` in `PivotTableChart.tsx`
(lines 297–335) flattens them, producing per-metric records that include `value` (the
metric value), `rows` and `columns` arrays describing the rollup level, and `__metricKey:
METRIC_KEY` to mark the metric pseudo-dimension.
3. `TableRenderer` passes these records into `PivotData` (TableRenderers.tsx lines
623–628), which constructs aggregators via the passthrough `cellValue` template
(utilities.ts lines 393–416) and feeds each record into `processRecord` (utilities.ts
lines 1117–1219). For rollup levels where the metric axis is the only dimension, the
metric-collapse logic (utilities.ts lines 1199–1218) mirrors each metric record into the
same total aggregators (`rowTotals[flatRowKey]`, `colTotals[flatColKey]`, and `allTotal`)
for a given key.
4. The `cellValue` aggregator used for these totals stores a single `val` and overwrites
it on every `push` (utilities.ts lines 397–400). When multiple metrics share the same
collapsed-total slot, their records are pushed sequentially and the last metric’s `value`
wins. Totals rows and the grand-total corner rendered by `renderTotalsRow` in
`TableRenderers.tsx` (lines 1444–1466) and `getAggregator([], [])` (utilities.ts lines
1222–1244) therefore show only the last metric’s value instead of a well-defined
cross-metric total whenever metrics are collapsed on one axis.
```
</details>
[](https://app.codeant.ai/fix-in-ide?tool=cursor&prompt_id=cbfb266c90994384b284f0088f2cd8c4&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset) [](https://app.codeant.ai/fix-in-ide?tool=vscode-claude&prompt_id=cbfb266c90994384b284f0088f2cd8c4&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset)
*(Use Cmd/Ctrl + Click for best experience)*
<details>
<summary><b>Prompt for AI Agent 🤖 </b></summary>
```mdx
This is a comment left during a code review.
**Path:** superset-frontend/plugins/plugin-chart-pivot-table/src/react-pivottable/utilities.ts
**Line:** 1207:1218
**Comment:**
*Logic Error: The metric-collapse path pushes multiple metric records into the same total aggregators, but the new passthrough aggregator stores only one value per cell, so totals for multi-metric pivots are overwritten by whichever metric is processed last. This produces incorrect row/column/grand totals whenever metrics are collapsed on one axis. You need a deterministic aggregation strategy for these totals (or explicitly suppress/disable them) instead of repeatedly overwriting the same slot.
Validate the correctness of the flagged issue. If correct, How can I resolve this? If you propose a fix, implement it and please make it concise.
Once fix is implemented, also check other comments on the same PR, and ask user if the user wants to fix the rest of the comments as well. if said yes, then fetch all the comments validate the correctness and implement a minimal fix
```
</details>
<a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F41184&comment_hash=09b91a556e09436da6a025207aa9a384b21749cf95c33e9ff63790f1ed4db60a&reaction=like'>👍</a> | <a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F41184&comment_hash=09b91a556e09436da6a025207aa9a384b21749cf95c33e9ff63790f1ed4db60a&reaction=dislike'>👎</a>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(table/pivot-table): correct non-additive totals/subtotals via DB rollup [POC for SIP-216] [superset]
Posted by "bito-code-review[bot] (via GitHub)" <gi...@apache.org>.
bito-code-review[bot] commented on PR #41184:
URL: https://github.com/apache/superset/pull/41184#issuecomment-4940850593
AI Code Review is in progress (usually takes 3 to 15 minutes unless it's a very large PR).
<details>
<summary>
Bito Usage Guide
</summary>
**Commands**
Type the following command in the pull request comment and save the comment.
- `/review` - Manually triggers a full AI review.
- `/pause` - Pauses automatic reviews on this pull request.
- `/resume` - Resumes automatic reviews.
- `/resolve` - Marks all Bito-posted review comments as resolved.
- `/abort` - Cancels all in-progress reviews.
Refer to the <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/available-commands" target="_blank" rel="noopener noreferrer">documentation</a> for additional commands.
**Configuration**
This repository uses `Superset` You can customize the agent settings <a href="https://alpha.bito.ai/home/ai-agents/code-review-agent" target="_blank" rel="noopener noreferrer">here</a> or contact your Bito workspace admin at evan@preset.io.
**Documentation & Help**
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/getting-started/install-run-using-bito-cloud/create-or-customize-an-agent-instance" target="_blank" rel="noopener noreferrer">Customize agent settings</a>
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/implementing-custom-code-review-rules" target="_blank" rel="noopener noreferrer">Review rules</a>
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent" target="_blank" rel="noopener noreferrer">General documentation</a>
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/faqs" target="_blank" rel="noopener noreferrer">FAQ</a>
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(table/pivot-table): correct non-additive totals/subtotals via DB rollup [POC for SIP-216] [superset]
Posted by "rusackas (via GitHub)" <gi...@apache.org>.
rusackas commented on code in PR #41184:
URL: https://github.com/apache/superset/pull/41184#discussion_r3565320217
##########
superset/common/grouping_sets.py:
##########
@@ -0,0 +1,116 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""
+SQL building blocks for the pivot-table non-additive totals optimization
+(SIP.md, phase 3b). When a datasource engine reports
+``supports_grouping_sets``, the N per-rollup-level queries can be collapsed into
+a single ``GROUPING SETS`` query: the database computes every level in one scan,
+and each returned row is attributed to its level via ``GROUPING()`` markers.
+
+These are the engine-agnostic SQL primitives. Wiring them into the query context
+(emitting one query and splitting the result back into per-level results) is the
+remaining integration; see SIP.md.
+"""
+
+from __future__ import annotations
+
+from collections.abc import Sequence
+
+import pandas as pd
+from sqlalchemy import func, tuple_
+from sqlalchemy.sql.elements import ColumnElement
+
+# Suffix for the per-column GROUPING() marker columns added to a GROUPING SETS
+# query. Chosen to be unlikely to collide with a real metric/column label.
+GROUPING_MARKER_SUFFIX = "__superset_grouping"
Review Comment:
Annotated it as `Final` in ef6691c.
##########
superset/common/query_context_processor.py:
##########
@@ -252,9 +256,60 @@ def get_query_result(self, query_object: QueryObject) -> QueryResult:
This method delegates to the datasource's get_query_result method,
which handles query execution, normalization, time offsets, and
post-processing.
+
+ When the query requests rollup ``grouping_sets`` but the engine does not
+ support native ``GROUPING SETS``, fall back to one query per level and
+ concatenate the results with ``GROUPING()``-equivalent markers, so the
+ combined result matches the shape the native path produces (SIP.md,
+ phase 3b). Engines that support it run the single native query.
"""
+ if query_object.grouping_sets and not self._supports_grouping_sets():
+ return self._grouping_sets_fallback(query_object)
return self._qc_datasource.get_query_result(query_object)
+ def _supports_grouping_sets(self) -> bool:
+ engine_spec: BaseEngineSpec | None = getattr(
+ self._qc_datasource, "db_engine_spec", None
+ )
+ return bool(engine_spec and engine_spec.supports_grouping_sets)
+
+ def _grouping_sets_fallback(self, query_object: QueryObject) -> QueryResult:
+ """
+ Emulate a GROUPING SETS query on engines without native support: run one
+ query per rollup level and concatenate, tagging each level's rows with
+ the same per-column markers the native path emits.
+ """
+ levels: list[list[str]] = query_object.grouping_sets
+ # Use the same label derivation as the native path (physical column name
+ # or adhoc column label) so both column kinds are represented and each
+ # label maps back to its own column, in the same order as the source
+ # list.
+ all_labels = [get_column_name(col) for col in query_object.columns]
Review Comment:
Added the `list[str]` annotation in ef6691c.
##########
superset/db_engine_specs/hive.py:
##########
@@ -96,6 +96,11 @@ class HiveEngineSpec(PrestoEngineSpec):
supports_dynamic_schema = True
supports_cross_catalog_queries = False
+ # Explicitly opt out (overriding the inherited PrestoEngineSpec value):
+ # Hive/Spark's GROUPING SETS + GROUPING() marker semantics have not been
+ # verified against this query pattern, so fall back to one query per
+ # rollup level instead of assuming native support.
+ supports_grouping_sets = False
Review Comment:
Annotated as `bool` in ef6691c.
##########
superset-frontend/plugins/plugin-chart-pivot-table/test/plugin/buildQuery.test.ts:
##########
@@ -56,9 +55,41 @@ const formData: PivotTableQueryFormData = {
currencyFormat: { symbol: 'USD', symbolPosition: 'prefix' },
};
-test('should build groupby with series in form data', () => {
+test('additive metrics use the fast-path: a single full-detail query', () => {
+ const { queries } = buildQuery({
+ ...formData,
+ metrics: [
+ {
+ expressionType: 'SIMPLE',
+ aggregate: 'SUM',
+ column: { column_name: 'num' },
+ label: 'sum_num',
+ },
+ ] as any,
+ });
Review Comment:
Typed the fixture as `QueryFormMetric[]` in ef6691c.
##########
superset-frontend/plugins/plugin-chart-pivot-table/test/plugin/buildQuery.test.ts:
##########
@@ -56,9 +55,41 @@ const formData: PivotTableQueryFormData = {
currencyFormat: { symbol: 'USD', symbolPosition: 'prefix' },
};
-test('should build groupby with series in form data', () => {
+test('additive metrics use the fast-path: a single full-detail query', () => {
+ const { queries } = buildQuery({
+ ...formData,
+ metrics: [
+ {
+ expressionType: 'SIMPLE',
+ aggregate: 'SUM',
+ column: { column_name: 'num' },
+ label: 'sum_num',
+ },
+ ] as any,
+ });
+ expect(queries).toHaveLength(1);
+ // The single leaf query carries all dimensions (2 rows + 2 cols).
+ expect(queries[0].columns).toHaveLength(4);
+});
+
+test('non-additive metrics emit a single GROUPING SETS query with all levels', () => {
+ // 2 row dims x 2 col dims -> (2+1) x (2+1) = 9 rollup levels, carried as
+ // grouping_sets on a single query (saved-metric strings are non-additive).
const queryContext = buildQuery(formData);
+ expect(queryContext.queries).toHaveLength(1);
const [query] = queryContext.queries;
+ // The single query selects the full set of dimensions ...
+ expect(query.columns).toHaveLength(4);
+ // ... and requests every rollup level via grouping_sets (grand total = []).
+ expect((query as any).grouping_sets).toHaveLength(9);
+ expect((query as any).grouping_sets[0]).toEqual([]);
+ expect((query as any).grouping_sets[8]).toHaveLength(4);
Review Comment:
Dropped the `any` — narrowed to a `QueryObject & { grouping_sets }` type in ef6691c.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(table/pivot-table): correct non-additive totals/subtotals via DB rollup [POC for SIP-216] [superset]
Posted by "codeant-ai-for-open-source[bot] (via GitHub)" <gi...@apache.org>.
codeant-ai-for-open-source[bot] commented on code in PR #41184:
URL: https://github.com/apache/superset/pull/41184#discussion_r3564137520
##########
superset-frontend/plugins/plugin-chart-pivot-table/src/PivotTableChart.tsx:
##########
@@ -341,22 +296,45 @@ export default function PivotTableChart(props: PivotTableProps) {
const unpivotedData = useMemo(
() =>
- data.reduce(
- (acc: Record<string, any>[], record: Record<string, any>) => [
- ...acc,
- ...metricNames
+ // `data` is now one entry per rollup level. Tag every record with the
+ // row/column dimension labels of the level that produced it (mirroring
+ // the METRIC_KEY injection used for the full rows/cols below) so
+ // PivotData can slot each pre-computed value without re-aggregating.
+ // buildGroupbyCombinations already applied transposePivot, so the level's
+ // groupby is display-oriented and is not transposed again here.
+ data.flatMap((query: QueryData) => {
+ let levelRows = query.groupby.rows.map(getColumnLabel);
+ let levelCols = query.groupby.columns.map(getColumnLabel);
+ if (metricsLayout === MetricsLayoutEnum.ROWS) {
+ levelRows = combineMetric
+ ? [...levelRows, METRIC_KEY]
+ : [METRIC_KEY, ...levelRows];
+ } else {
+ levelCols = combineMetric
+ ? [...levelCols, METRIC_KEY]
+ : [METRIC_KEY, ...levelCols];
+ }
+ return query.data.flatMap((record: Record<string, any>) =>
+ metricNames
.map((name: string) => ({
...record,
[METRIC_KEY]: name,
value: record[name],
// Mark currency column for per-cell currency detection in aggregators
__currencyColumn: currencyCodeColumn,
+ // The level this record belongs to (used by PivotData placement).
+ // Namespaced with a `__` prefix (like `__metricKey` below) so it
+ // can't collide with a real dataset column named `rows`/`columns`.
+ __rows: levelRows,
+ __columns: levelCols,
+ // Identify the metric pseudo-dimension so PivotData can feed the
+ // metric-collapsed totals (the opposite "Total" axis + corner).
+ __metricKey: METRIC_KEY,
}))
- .filter(record => record.value !== null),
- ],
- [],
- ),
- [data, metricNames, currencyCodeColumn],
+ .filter(r => r.value !== null),
Review Comment:
**Suggestion:** Filtering out every record whose metric value is `null` drops entire row/column keys for groups whose DB-computed value is null, so those groups disappear from the pivot instead of rendering as blank cells. Keep null-valued records and let the renderer/formatter display them as empty values. [incorrect condition logic]
<details>
<summary><b>Severity Level:</b> Major ⚠️</summary>
```mdx
- ❌ Pivot tables omit groups with null metric values.
- ⚠️ Totals/subtotals ignore groups whose metrics are null.
- ⚠️ Users see incomplete data for non-additive metrics.
```
</details>
<details>
<summary><b>Steps of Reproduction ✅ </b></summary>
```mdx
1. The backend returns rollup-level data for the pivot table as `QueryData[]`, where each
`QueryData` has `data: DataRecord[]` and `groupby` metadata
(`superset-frontend/plugins/plugin-chart-pivot-table/src/types.ts:61-68`).
`DataRecordValue` allows `null`, so a metric field (e.g. a ratio or CASE expression) can
legitimately be `null` for a given group.
2. In the frontend `PivotTableChart` component
(`superset-frontend/plugins/plugin-chart-pivot-table/src/PivotTableChart.tsx:181-215`),
the `unpivotedData` array is built in a `useMemo` at lines 297-337. Each backend row is
expanded per metric and tagged with `__rows`/`__columns`/`__metricKey` so the pivot engine
can place it correctly. The final step `.filter(r => r.value !== null)` at line 334 drops
every record whose metric value is `null`.
3. The resulting `unpivotedData` is passed to the `PivotTable` component
(`superset-frontend/plugins/plugin-chart-pivot-table/src/PivotTableChart.tsx:673-687`),
which is a thin wrapper around `TableRenderer` (`react-pivottable/PivotTable.tsx:20-28`,
`react-pivottable/TableRenderers.tsx:67-80`). `TableRenderer` constructs a `PivotData`
instance in `getBasePivotSettings` (`TableRenderers.tsx:105-112`), which iterates over
`props.data` via `PivotData.forEachRecord` (`react-pivottable/utilities.ts:216-224`) and
calls `processRecord` for each record.
4. `PivotData.processRecord` (`react-pivottable/utilities.ts:83-188`) registers row and
column keys and creates total/body aggregators based solely on the records it sees. If a
group (specific combination of `__rows` and `__columns`) only produced `null` metric
values from the database, all of its records were removed by `.filter(r => r.value !==
null)` and `processRecord` is never invoked for that row/column key. As a result,
`rowKeys`/`colKeys` do not contain that group, and the rendered table body
(`TableRenderers.tsx:188-191` iterating over `visibleRowKeys`) never shows that row/column
at all, instead of displaying the group headers with blank/empty metric cells. This
matches the suggestion: dropping null-valued records causes valid groups (with DB-computed
null metrics) to disappear entirely from the pivot.
```
</details>
[](https://app.codeant.ai/fix-in-ide?tool=cursor&prompt_id=073f1561f80c4f27943b31ff437c0d02&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset) [](https://app.codeant.ai/fix-in-ide?tool=vscode-claude&prompt_id=073f1561f80c4f27943b31ff437c0d02&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset)
*(Use Cmd/Ctrl + Click for best experience)*
<details>
<summary><b>Prompt for AI Agent 🤖 </b></summary>
```mdx
This is a comment left during a code review.
**Path:** superset-frontend/plugins/plugin-chart-pivot-table/src/PivotTableChart.tsx
**Line:** 334:334
**Comment:**
*Incorrect Condition Logic: Filtering out every record whose metric value is `null` drops entire row/column keys for groups whose DB-computed value is null, so those groups disappear from the pivot instead of rendering as blank cells. Keep null-valued records and let the renderer/formatter display them as empty values.
Validate the correctness of the flagged issue. If correct, How can I resolve this? If you propose a fix, implement it and please make it concise.
Once fix is implemented, also check other comments on the same PR, and ask user if the user wants to fix the rest of the comments as well. if said yes, then fetch all the comments validate the correctness and implement a minimal fix
```
</details>
<a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F41184&comment_hash=f639b371313f79b171e9149fdcc8e1501e93aa9f5f05c9217f846171b1a5c1b6&reaction=like'>👍</a> | <a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F41184&comment_hash=f639b371313f79b171e9149fdcc8e1501e93aa9f5f05c9217f846171b1a5c1b6&reaction=dislike'>👎</a>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(table/pivot-table): correct non-additive totals/subtotals via DB rollup [SIP-216] [superset]
Posted by "bito-code-review[bot] (via GitHub)" <gi...@apache.org>.
bito-code-review[bot] commented on PR #41184:
URL: https://github.com/apache/superset/pull/41184#issuecomment-5040326036
AI Code Review is in progress (usually takes 3 to 15 minutes unless it's a very large PR).
<details>
<summary>
Bito Usage Guide
</summary>
**Commands**
Type the following command in the pull request comment and save the comment.
- `/review` - Manually triggers a full AI review.
- `/pause` - Pauses automatic reviews on this pull request.
- `/resume` - Resumes automatic reviews.
- `/resolve` - Marks all Bito-posted review comments as resolved.
- `/abort` - Cancels all in-progress reviews.
Refer to the <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/available-commands" target="_blank" rel="noopener noreferrer">documentation</a> for additional commands.
**Configuration**
This repository uses `Superset` You can customize the agent settings <a href="https://alpha.bito.ai/home/ai-agents/code-review-agent" target="_blank" rel="noopener noreferrer">here</a> or contact your Bito workspace admin at evan@preset.io.
**Documentation & Help**
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/getting-started/install-run-using-bito-cloud/create-or-customize-an-agent-instance" target="_blank" rel="noopener noreferrer">Customize agent settings</a>
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/implementing-custom-code-review-rules" target="_blank" rel="noopener noreferrer">Review rules</a>
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent" target="_blank" rel="noopener noreferrer">General documentation</a>
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/faqs" target="_blank" rel="noopener noreferrer">FAQ</a>
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(table/pivot-table): correct non-additive totals/subtotals via DB rollup [SIP-216] [superset]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #41184:
URL: https://github.com/apache/superset/pull/41184#discussion_r3623679829
##########
superset/models/helpers.py:
##########
@@ -3962,7 +3996,10 @@ def get_sqla_query( # pylint: disable=too-many-arguments,too-many-locals,too-ma
direction = sa.asc if ascending else sa.desc
qry = qry.order_by(direction(col))
- if row_limit:
+ # A GROUPING SETS query computes a bounded set of rollup levels; applying
+ # a row LIMIT is both unnecessary and unparseable by some SQL parsers
+ # (LIMIT after a GROUPING SETS GROUP BY), so skip it for that path.
+ if row_limit and not use_grouping_sets:
qry = qry.limit(row_limit)
if row_offset:
qry = qry.offset(row_offset)
Review Comment:
`row_limit` is intentionally skipped for `GROUPING SETS` queries, which makes the Pivot Table “Cell limit” control ineffective for non-additive metrics and can significantly increase result size (and runtime/memory) compared to the previous leaf-only query. It also changes semantics vs additive fast-path where `row_limit` still applies.
Consider either (a) documenting/validating that `row_limit` is ignored when `grouping_sets` is used, or (b) implementing a leaf-only limiting strategy (likely requires splitting into per-level queries when a limit/offset is present, or another approach that preserves totals rows).
##########
superset/common/query_context_processor.py:
##########
@@ -252,9 +256,78 @@ def get_query_result(self, query_object: QueryObject) -> QueryResult:
This method delegates to the datasource's get_query_result method,
which handles query execution, normalization, time offsets, and
post-processing.
+
+ When the query requests rollup ``grouping_sets`` but the engine does not
+ support native ``GROUPING SETS``, fall back to one query per level and
+ concatenate the results with ``GROUPING()``-equivalent markers, so the
+ combined result matches the shape the native path produces (SIP.md,
+ phase 3b). Engines that support it run the single native query.
"""
+ if query_object.grouping_sets and not self._supports_grouping_sets():
+ return self._grouping_sets_fallback(query_object)
return self._qc_datasource.get_query_result(query_object)
+ def _supports_grouping_sets(self) -> bool:
+ engine_spec: BaseEngineSpec | None = getattr(
+ self._qc_datasource, "db_engine_spec", None
+ )
+ return bool(engine_spec and engine_spec.supports_grouping_sets)
+
+ def _grouping_sets_fallback(self, query_object: QueryObject) -> QueryResult:
+ """
+ Emulate a GROUPING SETS query on engines without native support: run one
+ query per rollup level and concatenate, tagging each level's rows with
+ the same per-column markers the native path emits.
+ """
+ levels: list[list[str]] = query_object.grouping_sets
+ # Use the same label derivation as the native path (physical column name
+ # or adhoc column label) so both column kinds are represented and each
+ # label maps back to its own column, in the same order as the source
+ # list.
+ all_labels: list[str] = [get_column_name(col) for col in query_object.columns]
+ label_to_column: dict[str, Column] = dict(
+ zip(all_labels, query_object.columns, strict=True)
+ )
+
+ frames: list[pd.DataFrame] = []
+ result: QueryResult | None = None
+ for level in levels:
+ level_labels: set[str] = set(level)
+ sub_query = copy.copy(query_object)
+ sub_query.grouping_sets = []
+ sub_query.columns = [
+ label_to_column[label] for label in all_labels if label in level_labels
+ ]
+ # A GROUPING SETS query computes a bounded set of rollup levels, so
+ # the native path never applies row_limit to it (see the
+ # `use_grouping_sets` check in models/helpers.py). Match that here:
+ # limiting each level's fallback sub-query independently would
+ # truncate subtotal/grand-total rows and diverge from the native
+ # result shape. The native path applies `row_offset` exactly once,
+ # to the combined multi-level result (see the unconditional
+ # `qry.offset()` call in models/helpers.py). Applying the same
+ # offset to each per-level sub-query independently would apply it
+ # once per level instead of once overall, and can silently drop
+ # low-row-count levels (e.g. the single grand-total row) entirely.
+ # Zero it here and apply it once after concatenation instead.
+ sub_query.row_limit = None
+ sub_query.row_offset = 0
+ result = self._qc_datasource.get_query_result(sub_query)
Review Comment:
The fallback explicitly clears `row_limit` for every per-level subquery, which (combined with the native path skipping LIMIT) means the Pivot Table “Cell limit” won’t constrain non-additive rollup queries on engines without native GROUPING SETS either. This can lead to unexpectedly large query results.
If the intent is to ignore limits for rollups, it’d be helpful to enforce/document this at the API/form-data layer; otherwise consider preserving/applying an equivalent limit behavior across levels.
##########
superset-frontend/plugins/plugin-chart-pivot-table/test/plugin/buildQuery.test.ts:
##########
@@ -17,18 +17,27 @@
* under the License.
*/
-import { TimeGranularity } from '@superset-ui/core';
+import {
+ QueryFormColumn,
+ QueryFormMetric,
+ QueryObject,
+ TimeGranularity,
+} from '@superset-ui/core';
import buildQuery from '../../src/plugin/buildQuery';
import { PivotTableQueryFormData } from '../../src/types';
+// buildQuery attaches `grouping_sets` (one entry per rollup level) to the
+// query object; it is not part of the base QueryObject type, so narrow to a
+// local type instead of casting through `any`.
+type GroupingSetsQuery = QueryObject & { grouping_sets: QueryFormColumn[][] };
Review Comment:
`grouping_sets` is serialized to the backend as `list[list[str]]` (column *labels*), not as `QueryFormColumn[][]`. Using `QueryFormColumn[][]` here makes the test type misleading and can hide future shape/type regressions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(table/pivot-table): correct non-additive totals/subtotals via DB rollup [POC for SIP-216] [superset]
Posted by "rusackas (via GitHub)" <gi...@apache.org>.
rusackas commented on code in PR #41184:
URL: https://github.com/apache/superset/pull/41184#discussion_r3566636681
##########
superset/common/grouping_sets.py:
##########
@@ -0,0 +1,117 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""
+SQL building blocks for the pivot-table non-additive totals optimization
+(SIP.md, phase 3b). When a datasource engine reports
+``supports_grouping_sets``, the N per-rollup-level queries can be collapsed into
+a single ``GROUPING SETS`` query: the database computes every level in one scan,
+and each returned row is attributed to its level via ``GROUPING()`` markers.
+
+These are the engine-agnostic SQL primitives. Wiring them into the query context
+(emitting one query and splitting the result back into per-level results) is the
+remaining integration; see SIP.md.
+"""
+
+from __future__ import annotations
+
+from collections.abc import Sequence
+from typing import Final
+
+import pandas as pd
+from sqlalchemy import func, tuple_
+from sqlalchemy.sql.elements import ColumnElement
+
+# Suffix for the per-column GROUPING() marker columns added to a GROUPING SETS
+# query. Chosen to be unlikely to collide with a real metric/column label.
+GROUPING_MARKER_SUFFIX: Final = "__superset_grouping"
+
+
+def grouping_marker_label(column_label: str) -> str:
+ """The output label of the GROUPING() marker for a groupby column."""
+ return f"{column_label}{GROUPING_MARKER_SUFFIX}"
+
+
+def grouping_sets_clause(
+ groups: Sequence[Sequence[ColumnElement]],
+) -> ColumnElement:
+ """
+ Build a ``GROUP BY GROUPING SETS (...)`` clause from rollup column groups.
+
+ Each group is the set of columns grouped at one rollup level; the empty
+ group ``()`` is the grand total. For example, groups ``[[a, b], [a], []]``
+ produce ``GROUPING SETS ((a, b), (a), ())``.
+
+ :param groups: one column list per rollup level
+ :return: a clause element suitable for ``select(...).group_by(...)``
+ """
+ return func.grouping_sets(*[tuple_(*group) for group in groups])
+
+
+def grouping_id_column(column: ColumnElement, label: str) -> ColumnElement:
+ """
+ Build a ``GROUPING(col) AS label`` marker column.
+
+ In a ``GROUPING SETS`` result, ``GROUPING(col)`` is ``0`` when ``col`` is
+ part of the row's grouping level and ``1`` when it has been rolled up
+ (aggregated away). Selecting one marker per groupby column lets the caller
+ attribute each returned row to its rollup level when splitting the single
+ result back into per-level results.
+
+ :param column: the groupby column to probe
+ :param label: the output label for the marker (see ``grouping_marker_label``)
+ :return: the labelled ``GROUPING(col)`` column
+ """
+ return func.grouping(column).label(label)
+
+
+def split_grouping_sets_result(
+ df: pd.DataFrame,
+ levels: Sequence[Sequence[str]],
+ groupby_columns: Sequence[str],
+) -> list[pd.DataFrame]:
+ """
+ Split a combined ``GROUPING SETS`` result into one DataFrame per rollup
+ level, the inverse of {@link grouping_sets_clause}.
+
+ Each row of ``df`` carries a ``GROUPING()`` marker per groupby column (named
+ by ``grouping_marker_label``): ``0`` if the column is grouped at that row's
+ level, ``1`` if it was rolled up. A row belongs to a level iff its markers
+ are ``0`` exactly for that level's columns. Marker columns are dropped from
+ the returned frames so each looks like an ordinary per-level query result.
+
+ :param df: the combined query result, including marker columns
+ :param levels: the grouped-column list for each rollup level (same order as
+ passed to ``grouping_sets_clause``)
+ :param groupby_columns: every groupby column that has a marker
+ :return: one DataFrame per level, in ``levels`` order
+ """
+ markers: list[str] = [grouping_marker_label(col) for col in groupby_columns]
+ results: list[pd.DataFrame] = []
+ for level in levels:
+ grouped: set[str] = set(level)
+ mask: pd.Series = pd.Series(True, index=df.index)
+ for col in groupby_columns:
+ expected = 0 if col in grouped else 1
Review Comment:
Added the type hint.
##########
superset/common/grouping_sets.py:
##########
@@ -0,0 +1,117 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""
+SQL building blocks for the pivot-table non-additive totals optimization
+(SIP.md, phase 3b). When a datasource engine reports
+``supports_grouping_sets``, the N per-rollup-level queries can be collapsed into
+a single ``GROUPING SETS`` query: the database computes every level in one scan,
+and each returned row is attributed to its level via ``GROUPING()`` markers.
+
+These are the engine-agnostic SQL primitives. Wiring them into the query context
+(emitting one query and splitting the result back into per-level results) is the
+remaining integration; see SIP.md.
+"""
+
+from __future__ import annotations
+
+from collections.abc import Sequence
+from typing import Final
+
+import pandas as pd
+from sqlalchemy import func, tuple_
+from sqlalchemy.sql.elements import ColumnElement
+
+# Suffix for the per-column GROUPING() marker columns added to a GROUPING SETS
+# query. Chosen to be unlikely to collide with a real metric/column label.
+GROUPING_MARKER_SUFFIX: Final = "__superset_grouping"
+
+
+def grouping_marker_label(column_label: str) -> str:
+ """The output label of the GROUPING() marker for a groupby column."""
+ return f"{column_label}{GROUPING_MARKER_SUFFIX}"
+
+
+def grouping_sets_clause(
+ groups: Sequence[Sequence[ColumnElement]],
+) -> ColumnElement:
+ """
+ Build a ``GROUP BY GROUPING SETS (...)`` clause from rollup column groups.
+
+ Each group is the set of columns grouped at one rollup level; the empty
+ group ``()`` is the grand total. For example, groups ``[[a, b], [a], []]``
+ produce ``GROUPING SETS ((a, b), (a), ())``.
+
+ :param groups: one column list per rollup level
+ :return: a clause element suitable for ``select(...).group_by(...)``
+ """
+ return func.grouping_sets(*[tuple_(*group) for group in groups])
+
+
+def grouping_id_column(column: ColumnElement, label: str) -> ColumnElement:
+ """
+ Build a ``GROUPING(col) AS label`` marker column.
+
+ In a ``GROUPING SETS`` result, ``GROUPING(col)`` is ``0`` when ``col`` is
+ part of the row's grouping level and ``1`` when it has been rolled up
+ (aggregated away). Selecting one marker per groupby column lets the caller
+ attribute each returned row to its rollup level when splitting the single
+ result back into per-level results.
+
+ :param column: the groupby column to probe
+ :param label: the output label for the marker (see ``grouping_marker_label``)
+ :return: the labelled ``GROUPING(col)`` column
+ """
+ return func.grouping(column).label(label)
+
+
+def split_grouping_sets_result(
+ df: pd.DataFrame,
+ levels: Sequence[Sequence[str]],
+ groupby_columns: Sequence[str],
+) -> list[pd.DataFrame]:
+ """
+ Split a combined ``GROUPING SETS`` result into one DataFrame per rollup
+ level, the inverse of {@link grouping_sets_clause}.
+
+ Each row of ``df`` carries a ``GROUPING()`` marker per groupby column (named
+ by ``grouping_marker_label``): ``0`` if the column is grouped at that row's
+ level, ``1`` if it was rolled up. A row belongs to a level iff its markers
+ are ``0`` exactly for that level's columns. Marker columns are dropped from
+ the returned frames so each looks like an ordinary per-level query result.
+
+ :param df: the combined query result, including marker columns
+ :param levels: the grouped-column list for each rollup level (same order as
+ passed to ``grouping_sets_clause``)
+ :param groupby_columns: every groupby column that has a marker
+ :return: one DataFrame per level, in ``levels`` order
+ """
+ markers: list[str] = [grouping_marker_label(col) for col in groupby_columns]
+ results: list[pd.DataFrame] = []
+ for level in levels:
+ grouped: set[str] = set(level)
+ mask: pd.Series = pd.Series(True, index=df.index)
+ for col in groupby_columns:
+ expected = 0 if col in grouped else 1
+ mask &= df[grouping_marker_label(col)] == expected
+ level_df = (
+ df[mask]
+ .drop(columns=[m for m in markers if m in df.columns])
+ .reset_index(drop=True)
+ )
Review Comment:
Added the type hint here too.
##########
superset-frontend/plugins/plugin-chart-pivot-table/test/plugin/utilities.test.ts:
##########
@@ -0,0 +1,446 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import { TimeGranularity } from '@superset-ui/core';
+import buildGroupbyCombinations, {
+ isAdditiveMetric,
+ allMetricsAdditive,
+ additiveReducerFor,
+ synthesizeAdditiveLevels,
+ splitGroupingSetsResult,
+ groupingMarkerLabel,
+} from '../../src/plugin/utilities';
+import { PivotTableQueryFormData, MetricsLayoutEnum } from '../../src/types';
+
+const baseFormData = {
+ groupbyRows: ['row1', 'row2'],
+ groupbyColumns: ['col1', 'col2'],
+ metrics: ['metric1', 'metric2'],
+ tableRenderer: 'Table With Subtotal',
+ colOrder: 'key_a_to_z',
+ rowOrder: 'key_a_to_z',
+ aggregateFunction: 'Sum',
+ metricsLayout: MetricsLayoutEnum.ROWS,
+ transposePivot: false,
+ rowSubtotalPosition: true,
+ colSubtotalPosition: true,
+ colTotals: true,
+ colSubTotals: true,
+ rowTotals: true,
+ rowSubTotals: true,
+ valueFormat: 'SMART_NUMBER',
+ datasource: '5__table',
+ viz_type: 'my_chart',
+ width: 800,
+ height: 600,
+ combineMetric: false,
+ verboseMap: {},
+ columnFormats: {},
+ currencyFormats: {},
+ metricColorFormatters: [],
+ dateFormatters: {},
+ setDataMask: () => {},
+ legacy_order_by: 'count',
+ order_desc: true,
+ margin: 0,
+ time_grain_sqla: TimeGranularity.MONTH,
+ temporal_columns_lookup: { col1: true },
+ currencyFormat: { symbol: 'USD', symbolPosition: 'prefix' },
+} as unknown as PivotTableQueryFormData;
+
+test('should build all combinations for basic pivot table', () => {
+ const combinations = buildGroupbyCombinations(baseFormData);
+
+ expect(combinations).toEqual([
+ { rows: [], columns: [] },
+ { rows: [], columns: ['col1'] },
+ { rows: [], columns: ['col1', 'col2'] },
+ { rows: ['row1'], columns: [] },
+ { rows: ['row1'], columns: ['col1'] },
+ { rows: ['row1'], columns: ['col1', 'col2'] },
+ { rows: ['row1', 'row2'], columns: [] },
+ { rows: ['row1', 'row2'], columns: ['col1'] },
+ { rows: ['row1', 'row2'], columns: ['col1', 'col2'] },
+ ]);
+
+ expect(combinations).toHaveLength(9);
+});
+
+test('should handle transposed pivot correctly', () => {
+ const modifiedFormData = {
+ ...baseFormData,
+ transposePivot: true,
+ };
+
+ const combinations = buildGroupbyCombinations(modifiedFormData);
+
+ expect(combinations).toEqual([
+ { rows: [], columns: [] },
+ { rows: [], columns: ['row1'] },
+ { rows: [], columns: ['row1', 'row2'] },
+ { rows: ['col1'], columns: [] },
+ { rows: ['col1'], columns: ['row1'] },
+ { rows: ['col1'], columns: ['row1', 'row2'] },
+ { rows: ['col1', 'col2'], columns: [] },
+ { rows: ['col1', 'col2'], columns: ['row1'] },
+ { rows: ['col1', 'col2'], columns: ['row1', 'row2'] },
+ ]);
+});
+
+test('should filter combinations when combineMetric is true with ROWS layout', () => {
+ const modifiedFormData = {
+ ...baseFormData,
+ combineMetric: true,
+ metricsLayout: MetricsLayoutEnum.ROWS,
+ };
+
+ const combinations = buildGroupbyCombinations(modifiedFormData);
+
+ expect(combinations).toEqual([
+ { rows: ['row1', 'row2'], columns: [] },
+ { rows: ['row1', 'row2'], columns: ['col1'] },
+ { rows: ['row1', 'row2'], columns: ['col1', 'col2'] },
+ ]);
+
+ expect(combinations).toHaveLength(3);
+});
+
+test('should filter combinations when combineMetric is true with COLUMNS layout', () => {
+ const modifiedFormData = {
+ ...baseFormData,
+ combineMetric: true,
+ metricsLayout: MetricsLayoutEnum.COLUMNS,
+ };
+
+ const combinations = buildGroupbyCombinations(modifiedFormData);
+
+ expect(combinations).toEqual([
+ { rows: [], columns: ['col1', 'col2'] },
+ { rows: ['row1'], columns: ['col1', 'col2'] },
+ { rows: ['row1', 'row2'], columns: ['col1', 'col2'] },
+ ]);
+
+ expect(combinations).toHaveLength(3);
+});
+
+test('should handle single dimension in rows only', () => {
+ const modifiedFormData = {
+ ...baseFormData,
+ groupbyRows: ['row'],
+ groupbyColumns: [],
+ };
+
+ const combinations = buildGroupbyCombinations(modifiedFormData);
+
+ expect(combinations).toEqual([
+ { rows: [], columns: [] },
+ { rows: ['row'], columns: [] },
+ ]);
+
+ expect(combinations).toHaveLength(2);
+});
+
+test('should handle single dimension in columns only', () => {
+ const modifiedFormData = {
+ ...baseFormData,
+ groupbyRows: [],
+ groupbyColumns: ['col'],
+ };
+
+ const combinations = buildGroupbyCombinations(modifiedFormData);
+
+ expect(combinations).toEqual([
+ { rows: [], columns: [] },
+ { rows: [], columns: ['col'] },
+ ]);
+
+ expect(combinations).toHaveLength(2);
+});
+
+test('should handle empty groupby arrays', () => {
+ const modifiedFormData = {
+ ...baseFormData,
+ groupbyRows: [],
+ groupbyColumns: [],
+ };
+
+ const combinations = buildGroupbyCombinations(modifiedFormData);
+
+ expect(combinations).toEqual([{ rows: [], columns: [] }]);
+
+ expect(combinations).toHaveLength(1);
+});
+
+test('should work with combineMetric and transposed pivot', () => {
+ const modifiedFormData = {
+ ...baseFormData,
+ transposePivot: true,
+ combineMetric: true,
+ metricsLayout: MetricsLayoutEnum.COLUMNS,
+ };
+
+ const combinations = buildGroupbyCombinations(modifiedFormData);
+
+ expect(combinations).toEqual([
+ { rows: [], columns: ['row1', 'row2'] },
+ { rows: ['col1'], columns: ['row1', 'row2'] },
+ { rows: ['col1', 'col2'], columns: ['row1', 'row2'] },
+ ]);
+});
+
+test('should handle combineMetric with empty arrays correctly', () => {
+ const modifiedFormData = {
+ ...baseFormData,
+ groupbyRows: [],
+ groupbyColumns: ['col'],
+ combineMetric: true,
+ metricsLayout: MetricsLayoutEnum.ROWS,
+ };
+
+ const combinations = buildGroupbyCombinations(modifiedFormData);
+
+ expect(combinations).toEqual([
+ { rows: [], columns: [] },
+ { rows: [], columns: ['col'] },
+ ]);
+});
+
+test('should work with large number of dimensions', () => {
+ const modifiedFormData = {
+ ...baseFormData,
+ groupbyRows: ['r1', 'r2', 'r3', 'r4'],
+ groupbyColumns: ['c1', 'c2', 'c3'],
+ };
+
+ const combinations = buildGroupbyCombinations(modifiedFormData);
+
+ expect(combinations).toHaveLength(20);
+
+ expect(combinations).toContainEqual({ rows: [], columns: [] });
+ expect(combinations).toContainEqual({
+ rows: ['r1', 'r2', 'r3', 'r4'],
+ columns: ['c1', 'c2', 'c3'],
+ });
+});
+
+test('isAdditiveMetric: SIMPLE metrics with additive aggregates are additive', () => {
+ expect(
+ isAdditiveMetric({
+ expressionType: 'SIMPLE',
+ aggregate: 'SUM',
+ column: { column_name: 'num' },
+ label: 'sum_num',
+ } as any),
+ ).toBe(true);
+ expect(
+ isAdditiveMetric({
+ expressionType: 'SIMPLE',
+ aggregate: 'COUNT',
+ column: { column_name: 'num' },
+ label: 'count_num',
+ } as any),
+ ).toBe(true);
+});
+
+test('isAdditiveMetric: non-additive aggregates, SQL, and saved metrics are not additive', () => {
+ expect(
+ isAdditiveMetric({
+ expressionType: 'SIMPLE',
+ aggregate: 'AVG',
+ column: { column_name: 'num' },
+ label: 'avg_num',
+ } as any),
+ ).toBe(false);
Review Comment:
Swapped the `as any` casts in this file for `QueryFormMetric`.
##########
superset/common/query_context_processor.py:
##########
@@ -252,9 +256,69 @@ def get_query_result(self, query_object: QueryObject) -> QueryResult:
This method delegates to the datasource's get_query_result method,
which handles query execution, normalization, time offsets, and
post-processing.
+
+ When the query requests rollup ``grouping_sets`` but the engine does not
+ support native ``GROUPING SETS``, fall back to one query per level and
+ concatenate the results with ``GROUPING()``-equivalent markers, so the
+ combined result matches the shape the native path produces (SIP.md,
+ phase 3b). Engines that support it run the single native query.
"""
+ if query_object.grouping_sets and not self._supports_grouping_sets():
+ return self._grouping_sets_fallback(query_object)
return self._qc_datasource.get_query_result(query_object)
+ def _supports_grouping_sets(self) -> bool:
+ engine_spec: BaseEngineSpec | None = getattr(
+ self._qc_datasource, "db_engine_spec", None
+ )
+ return bool(engine_spec and engine_spec.supports_grouping_sets)
+
+ def _grouping_sets_fallback(self, query_object: QueryObject) -> QueryResult:
+ """
+ Emulate a GROUPING SETS query on engines without native support: run one
+ query per rollup level and concatenate, tagging each level's rows with
+ the same per-column markers the native path emits.
+ """
+ levels: list[list[str]] = query_object.grouping_sets
+ # Use the same label derivation as the native path (physical column name
+ # or adhoc column label) so both column kinds are represented and each
+ # label maps back to its own column, in the same order as the source
+ # list.
+ all_labels: list[str] = [get_column_name(col) for col in query_object.columns]
+ label_to_column: dict[str, Column] = dict(
+ zip(all_labels, query_object.columns, strict=True)
+ )
+
+ frames: list[pd.DataFrame] = []
+ result: QueryResult | None = None
+ for level in levels:
+ level_labels: set[str] = set(level)
+ sub_query = copy.copy(query_object)
+ sub_query.grouping_sets = []
+ sub_query.columns = [
+ label_to_column[label] for label in all_labels if label in level_labels
+ ]
+ # A GROUPING SETS query computes a bounded set of rollup levels, so
+ # the native path never applies row_limit to it (see the
+ # `use_grouping_sets` check in models/helpers.py). Match that here:
+ # limiting each level's fallback sub-query independently would
+ # truncate subtotal/grand-total rows and diverge from the native
+ # result shape.
+ sub_query.row_limit = None
Review Comment:
Good catch. Zeroed row_offset on each per-level sub-query and applied it once after concatenating, so it matches the single-offset semantics of the native GROUPING SETS path instead of offsetting every level independently.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(table/pivot-table): correct non-additive totals/subtotals via DB rollup [POC for SIP-216] [superset]
Posted by "bito-code-review[bot] (via GitHub)" <gi...@apache.org>.
bito-code-review[bot] commented on code in PR #41184:
URL: https://github.com/apache/superset/pull/41184#discussion_r3556438821
##########
superset-frontend/plugins/plugin-chart-pivot-table/test/react-pivottable/tableRenders.test.tsx:
##########
@@ -103,35 +150,41 @@ test('TableRenderer renders row headers from pivot data', () => {
});
test('TableRenderer renders aggregated cell values', () => {
- const props = buildDefaultProps();
+ const props = buildDefaultProps({
+ data: TAGGED_COUNT_DATA,
+ vals: ['value'],
+ });
renderWithTheme(<TableRenderer {...props} />);
- // With "Count" aggregator, each cell (row x col intersection) should
- // contain "1" because each combination appears exactly once.
+ // Each leaf cell (row x col intersection) holds the DB-computed value "1".
const cells = screen.getAllByRole('gridcell');
const cellTexts = cells.map(cell => cell.textContent);
- // There should be cell values of "1" for each of the four intersections
- // (blue+circle, blue+square, red+circle, red+square).
- const onesCount = cellTexts.filter(text => text === '1').length;
+ // There should be a "1" leaf cell for each of the four intersections
+ // (blue+circle, blue+square, red+circle, red+square). The default formatter
+ // renders with two decimals in this test (production applies the metric's
+ // own format).
+ const onesCount = cellTexts.filter(text => text === '1.00').length;
expect(onesCount).toBeGreaterThanOrEqual(4);
});
test('TableRenderer renders row totals when rowTotals is enabled', () => {
const props = buildDefaultProps({
+ data: TAGGED_COUNT_DATA,
+ vals: ['value'],
tableOptions: { rowTotals: true, colTotals: true },
});
renderWithTheme(<TableRenderer {...props} />);
- // Row totals column should show "2" for each color (blue has 2 records,
- // red has 2 records).
+ // Row totals column should show the DB-computed "2" for each color (blue has
+ // 2 records, red has 2 records).
const totalCells = screen
.getAllByRole('gridcell')
.filter(cell => cell.classList.contains('pvtTotal'));
expect(totalCells.length).toBeGreaterThan(0);
const totalValues = totalCells.map(cell => cell.textContent);
- expect(totalValues).toContain('2');
+ expect(totalValues).toContain('2.00');
Review Comment:
<div>
<div id="suggestion">
<div id="issue"><b>Count aggregator assertion mismatch</b></div>
<div id="fix">
Same formatter mismatch as line 167: count aggregator uses usFmtInt (0 decimals), but assertion expects '2.00' (2 decimals).
</div>
</div>
<small><i>Code Review Run #275caa</i></small>
</div>
---
Should Bito avoid suggestions like this for future reviews? (<a href=https://alpha.bito.ai/home/ai-agents/review-rules>Manage Rules</a>)
- [ ] Yes, avoid them
##########
superset-frontend/plugins/plugin-chart-pivot-table/test/react-pivottable/tableRenders.test.tsx:
##########
@@ -103,35 +150,41 @@ test('TableRenderer renders row headers from pivot data', () => {
});
test('TableRenderer renders aggregated cell values', () => {
- const props = buildDefaultProps();
+ const props = buildDefaultProps({
+ data: TAGGED_COUNT_DATA,
+ vals: ['value'],
+ });
renderWithTheme(<TableRenderer {...props} />);
- // With "Count" aggregator, each cell (row x col intersection) should
- // contain "1" because each combination appears exactly once.
+ // Each leaf cell (row x col intersection) holds the DB-computed value "1".
const cells = screen.getAllByRole('gridcell');
const cellTexts = cells.map(cell => cell.textContent);
- // There should be cell values of "1" for each of the four intersections
- // (blue+circle, blue+square, red+circle, red+square).
- const onesCount = cellTexts.filter(text => text === '1').length;
+ // There should be a "1" leaf cell for each of the four intersections
+ // (blue+circle, blue+square, red+circle, red+square). The default formatter
+ // renders with two decimals in this test (production applies the metric's
+ // own format).
+ const onesCount = cellTexts.filter(text => text === '1.00').length;
Review Comment:
<div>
<div id="suggestion">
<div id="issue"><b>Count aggregator assertion mismatch</b></div>
<div id="fix">
The test's `aggregatorsFactory` passes `aggregatorTemplates.count()` without a formatter, so it defaults to `usFmtInt` (0 decimal places). The assertion expects `'1.00'` but count will produce `'1'`. The same issue appears on line 187 for `'2.00'` vs `'2'`.
</div>
</div>
<small><i>Code Review Run #275caa</i></small>
</div>
---
Should Bito avoid suggestions like this for future reviews? (<a href=https://alpha.bito.ai/home/ai-agents/review-rules>Manage Rules</a>)
- [ ] Yes, avoid them
##########
superset-frontend/plugins/plugin-chart-pivot-table/src/react-pivottable/utilities.ts:
##########
@@ -930,14 +962,10 @@ class PivotData {
'PivotData',
);
- const aggregatorsFactory = this.props.aggregatorsFactory as (
- fmt: unknown,
- ) => Record<string, (vals: unknown) => (...args: unknown[]) => Aggregator>;
- const aggregatorName = this.props.aggregatorName as string;
const vals = this.props.vals as string[];
- this.aggregator = aggregatorsFactory(this.props.defaultFormatter)[
- aggregatorName
- ](vals);
+ // Values come pre-aggregated from the database (one query per rollup level),
+ // so the pivot stores them verbatim via `cellValue` instead of aggregating.
+ this.aggregator = cellValue(this.props.defaultFormatter as Formatter)(vals);
Review Comment:
<div>
<div id="suggestion">
<div id="issue"><b>Missing function export</b></div>
<div id="fix">
The new `cellValue` aggregator is used internally but not exported. This breaks consistency with `aggregators` and `aggregatorTemplates` which are exported. Consider exporting it alongside the other aggregators for parity and easier unit testing.
</div>
</div>
<small><i>Code Review Run #275caa</i></small>
</div>
---
Should Bito avoid suggestions like this for future reviews? (<a href=https://alpha.bito.ai/home/ai-agents/review-rules>Manage Rules</a>)
- [ ] Yes, avoid them
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(table/pivot-table): correct non-additive totals/subtotals via DB rollup [POC for SIP-216] [superset]
Posted by "codeant-ai-for-open-source[bot] (via GitHub)" <gi...@apache.org>.
codeant-ai-for-open-source[bot] commented on code in PR #41184:
URL: https://github.com/apache/superset/pull/41184#discussion_r3565566152
##########
superset-frontend/plugins/plugin-chart-pivot-table/src/plugin/transformProps.ts:
##########
@@ -88,20 +98,63 @@ export default function transformProps(chartProps: ChartProps<QueryFormData>) {
emitCrossFilters,
theme,
} = chartProps;
- const {
- data,
- colnames,
- coltypes,
- detected_currency: detectedCurrency,
- } = queriesData[0];
+ const groupbyCombinations = buildGroupbyCombinations(
+ formData as PivotTableQueryFormData,
+ );
+ const metricsArr = ensureIsArray(formData.metrics);
+ let data: QueryData[];
+ if (allMetricsAdditive(metricsArr)) {
+ // Additive fast-path: a single full-detail query was issued; synthesize
+ // each rollup level by reducing the leaf rows on the client (see SIP.md).
+ const leafRows = queriesData[0].data;
+ const metricReducers: Record<string, RollupReducer> = {};
+ metricsArr.forEach(metric => {
+ metricReducers[getMetricLabel(metric)] = additiveReducerFor(metric);
+ });
+ const labelLevels = groupbyCombinations.map(combination => ({
+ rows: combination.rows.map(getColumnLabel),
+ columns: combination.columns.map(getColumnLabel),
+ }));
+ const synthesized = synthesizeAdditiveLevels(
+ leafRows,
+ labelLevels,
+ metricReducers,
+ );
Review Comment:
**Suggestion:** The new additive fast-path is enabled for MIN/MAX metrics too, but the reducer pipeline it calls coerces metric values to numbers; MIN/MAX over non-numeric data (for example strings or temporal values) will therefore synthesize wrong/null totals. Restrict the fast-path to numeric-safe aggregates or preserve native value types when reducing MIN/MAX. [type error]
<details>
<summary><b>Severity Level:</b> Major ⚠️</summary>
```mdx
- ❌ MIN/MAX rollup totals null for temporal/string metrics.
- ⚠️ Pivot table diverges from backend MIN/MAX semantics.
```
</details>
<details>
<summary><b>Steps of Reproduction ✅ </b></summary>
```mdx
1. Create a Pivot Table chart whose metrics include a MIN or MAX over a non-numeric
column, for example `MIN(event_time)` on a temporal column; the chart form data is sent to
the frontend as `formData.metrics` in `ChartProps` consumed by `transformProps` at
`superset-frontend/plugins/plugin-chart-pivot-table/src/plugin/transformProps.ts:54-100`.
2. In `transformProps`, `metricsArr` is constructed with `ensureIsArray(formData.metrics)`
at line 104, and `allMetricsAdditive(metricsArr)` at line 106 calls `isAdditiveMetric` in
`superset-frontend/plugins/plugin-chart-pivot-table/src/plugin/utilities.ts:36-41`, which
treats SIMPLE metrics with aggregate `MIN` or `MAX` as additive regardless of the
underlying value type; thus `allMetricsAdditive` returns true when all metrics are
SUM/COUNT/MIN/MAX, including MIN/MAX over temporal or string columns.
3. Because `allMetricsAdditive(metricsArr)` is true, the additive fast-path block at
`transformProps.ts:106-123` runs: it builds `metricReducers` using
`additiveReducerFor(metric)` (utilities.ts:58-63, returning `'min'`/`'max'`), constructs
`labelLevels`, and calls `synthesizeAdditiveLevels(leafRows, labelLevels, metricReducers)`
(utilities.ts:83-123) to synthesize rollup rows from the full-detail
`queriesData[0].data`.
4. Inside `synthesizeAdditiveLevels` (`plugin/utilities.ts:111-119`), each metric value is
converted with `Number(v)` and filtered so only entries where `Number(v)` is not `NaN` are
kept; for MIN/MAX metrics over temporal or string values, every value fails this numeric
check and `values` becomes empty, so `out[metricKey]` is set to `null` for all
totals/subtotals. The resulting `data: QueryData[]` (built at `transformProps.ts:123-126`)
is passed to `PivotTableChart`, which then renders rollup-level cells for these MIN/MAX
metrics as null/blank, even though the backend could compute correct MIN/MAX totals over
non-numeric types via GROUPING SETS.
```
</details>
[](https://app.codeant.ai/fix-in-ide?tool=cursor&prompt_id=b9e3ec7e855b43e68f9fb8a941e696f6&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset) [](https://app.codeant.ai/fix-in-ide?tool=vscode-claude&prompt_id=b9e3ec7e855b43e68f9fb8a941e696f6&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset)
*(Use Cmd/Ctrl + Click for best experience)*
<details>
<summary><b>Prompt for AI Agent 🤖 </b></summary>
```mdx
This is a comment left during a code review.
**Path:** superset-frontend/plugins/plugin-chart-pivot-table/src/plugin/transformProps.ts
**Line:** 106:122
**Comment:**
*Type Error: The new additive fast-path is enabled for MIN/MAX metrics too, but the reducer pipeline it calls coerces metric values to numbers; MIN/MAX over non-numeric data (for example strings or temporal values) will therefore synthesize wrong/null totals. Restrict the fast-path to numeric-safe aggregates or preserve native value types when reducing MIN/MAX.
Validate the correctness of the flagged issue. If correct, How can I resolve this? If you propose a fix, implement it and please make it concise.
Once fix is implemented, also check other comments on the same PR, and ask user if the user wants to fix the rest of the comments as well. if said yes, then fetch all the comments validate the correctness and implement a minimal fix
```
</details>
<a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F41184&comment_hash=7c80a70dad5f4c0ba97ecb84eedf4ebf2638d53c7289ad60fad18c03a0afb521&reaction=like'>👍</a> | <a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F41184&comment_hash=7c80a70dad5f4c0ba97ecb84eedf4ebf2638d53c7289ad60fad18c03a0afb521&reaction=dislike'>👎</a>
##########
superset-frontend/plugins/plugin-chart-pivot-table/src/PivotTableChart.tsx:
##########
@@ -383,22 +338,45 @@ export default function PivotTableChart(props: PivotTableProps) {
const unpivotedData = useMemo(
() =>
- data.reduce(
- (acc: DataRecord[], record: DataRecord) => [
- ...acc,
- ...metricNames
+ // `data` is now one entry per rollup level. Tag every record with the
+ // row/column dimension labels of the level that produced it (mirroring
+ // the METRIC_KEY injection used for the full rows/cols below) so
+ // PivotData can slot each pre-computed value without re-aggregating.
+ // buildGroupbyCombinations already applied transposePivot, so the level's
+ // groupby is display-oriented and is not transposed again here.
+ data.flatMap((query: QueryData) => {
+ let levelRows = query.groupby.rows.map(getColumnLabel);
+ let levelCols = query.groupby.columns.map(getColumnLabel);
+ if (metricsLayout === MetricsLayoutEnum.ROWS) {
+ levelRows = combineMetric
+ ? [...levelRows, METRIC_KEY]
+ : [METRIC_KEY, ...levelRows];
+ } else {
+ levelCols = combineMetric
+ ? [...levelCols, METRIC_KEY]
+ : [METRIC_KEY, ...levelCols];
+ }
+ return query.data.flatMap((record: DataRecord) =>
+ metricNames
.map((name: string) => ({
...record,
[METRIC_KEY]: name,
value: record[name],
// Mark currency column for per-cell currency detection in aggregators
__currencyColumn: currencyCodeColumn,
+ // The level this record belongs to (used by PivotData placement).
+ // Namespaced with a `__` prefix (like `__metricKey` below) so it
+ // can't collide with a real dataset column named `rows`/`columns`.
+ __rows: levelRows,
+ __columns: levelCols,
+ // Identify the metric pseudo-dimension so PivotData can feed the
+ // metric-collapsed totals (the opposite "Total" axis + corner).
+ __metricKey: METRIC_KEY,
}))
- .filter(record => record.value !== null),
- ],
- [],
- ),
- [data, metricNames, currencyCodeColumn],
+ .filter(r => r.value !== null),
Review Comment:
**Suggestion:** Filtering out records where metric value is null drops those rows from pivot ingestion entirely, so dimension members with null metrics disappear instead of rendering as empty cells. Keep null-valued records and let the cell formatter render blank/null output so row/column headers and totals remain complete. [logic error]
<details>
<summary><b>Severity Level:</b> Major ⚠️</summary>
```mdx
- ❌ Pivot table hides groups with all-null metric values.
- ⚠️ Totals/subtotals omit null-only dimension members.
```
</details>
<details>
<summary><b>Steps of Reproduction ✅ </b></summary>
```mdx
1. Configure a Pivot Table chart in Superset with any groupby rows/columns and a metric
that can legitimately evaluate to NULL for some dimension combinations (e.g. a ratio
metric or a MIN/MAX over sparse data), then run the chart so the backend returns grouped
data to the pivot plugin (handled by `transformProps` at
`superset-frontend/plugins/plugin-chart-pivot-table/src/plugin/transformProps.ts:54-144`).
2. On the client, `transformProps` builds the rollup-level `data: QueryData[]` and passes
it as props into `PivotTableChart` (constructor at
`superset-frontend/plugins/plugin-chart-pivot-table/src/PivotTableChart.tsx:223-257`),
where each `QueryData` entry contains rows for one rollup level, including rows whose
metric values are `null`.
3. Inside `PivotTableChart`, the `unpivotedData` memo at `PivotTableChart.tsx:339-377`
expands each `QueryData` into per-metric records, assigning `value: record[name]` and
tagging the rollup level with `__rows`/`__columns`/`__metricKey` (lines 360-374), then
applies `.filter(r => r.value !== null)` at line 376; any record where the metric value is
`null` is removed entirely from `unpivotedData`.
4. The `PivotTable` component (from `./react-pivottable`) feeds `unpivotedData` into
`PivotData.processRecord` (see
`superset-frontend/plugins/plugin-chart-pivot-table/src/react-pivottable/utilities.ts:1119-1223`),
which creates rowKeys/colKeys only for dimension combinations that have at least one
record; combinations whose metric is `null` for all metrics now have no records at all and
therefore no rowKey/colKey, so the corresponding row/column headers disappear from the
rendered pivot instead of showing empty/null cells.
```
</details>
[](https://app.codeant.ai/fix-in-ide?tool=cursor&prompt_id=812b9d9d27a04f2ab1e1c7a35848a18b&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset) [](https://app.codeant.ai/fix-in-ide?tool=vscode-claude&prompt_id=812b9d9d27a04f2ab1e1c7a35848a18b&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset)
*(Use Cmd/Ctrl + Click for best experience)*
<details>
<summary><b>Prompt for AI Agent 🤖 </b></summary>
```mdx
This is a comment left during a code review.
**Path:** superset-frontend/plugins/plugin-chart-pivot-table/src/PivotTableChart.tsx
**Line:** 376:376
**Comment:**
*Logic Error: Filtering out records where metric value is null drops those rows from pivot ingestion entirely, so dimension members with null metrics disappear instead of rendering as empty cells. Keep null-valued records and let the cell formatter render blank/null output so row/column headers and totals remain complete.
Validate the correctness of the flagged issue. If correct, How can I resolve this? If you propose a fix, implement it and please make it concise.
Once fix is implemented, also check other comments on the same PR, and ask user if the user wants to fix the rest of the comments as well. if said yes, then fetch all the comments validate the correctness and implement a minimal fix
```
</details>
<a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F41184&comment_hash=71f9ec7cb498c02e03979057750ff1c08d1028f181894d59a1f94be99031c6cb&reaction=like'>👍</a> | <a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F41184&comment_hash=71f9ec7cb498c02e03979057750ff1c08d1028f181894d59a1f94be99031c6cb&reaction=dislike'>👎</a>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(table/pivot-table): correct non-additive totals/subtotals via DB rollup [POC for SIP-216] [superset]
Posted by "bito-code-review[bot] (via GitHub)" <gi...@apache.org>.
bito-code-review[bot] commented on PR #41184:
URL: https://github.com/apache/superset/pull/41184#issuecomment-4950297767
**Bito Automatic Review Failed - Technical Failure** <br/><br/> Bito encountered technical difficulties while generating code feedback . To retry, type `/review` in a comment and save. If the issue persists, contact support@bito.ai and provide the following details: <br/><br/>Agent Run ID: baaf0299-9adc-4bfa-ab87-dfb0644195c5
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(table/pivot-table): correct non-additive totals/subtotals via DB rollup [POC for SIP-216] [superset]
Posted by "codeant-ai-for-open-source[bot] (via GitHub)" <gi...@apache.org>.
codeant-ai-for-open-source[bot] commented on code in PR #41184:
URL: https://github.com/apache/superset/pull/41184#discussion_r3565563447
##########
superset/common/grouping_sets.py:
##########
@@ -0,0 +1,117 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""
+SQL building blocks for the pivot-table non-additive totals optimization
+(SIP.md, phase 3b). When a datasource engine reports
+``supports_grouping_sets``, the N per-rollup-level queries can be collapsed into
+a single ``GROUPING SETS`` query: the database computes every level in one scan,
+and each returned row is attributed to its level via ``GROUPING()`` markers.
+
+These are the engine-agnostic SQL primitives. Wiring them into the query context
+(emitting one query and splitting the result back into per-level results) is the
+remaining integration; see SIP.md.
+"""
+
+from __future__ import annotations
+
+from collections.abc import Sequence
+from typing import Final
+
+import pandas as pd
+from sqlalchemy import func, tuple_
+from sqlalchemy.sql.elements import ColumnElement
+
+# Suffix for the per-column GROUPING() marker columns added to a GROUPING SETS
+# query. Chosen to be unlikely to collide with a real metric/column label.
+GROUPING_MARKER_SUFFIX: Final = "__superset_grouping"
+
+
+def grouping_marker_label(column_label: str) -> str:
+ """The output label of the GROUPING() marker for a groupby column."""
+ return f"{column_label}{GROUPING_MARKER_SUFFIX}"
+
+
+def grouping_sets_clause(
+ groups: Sequence[Sequence[ColumnElement]],
+) -> ColumnElement:
+ """
+ Build a ``GROUP BY GROUPING SETS (...)`` clause from rollup column groups.
+
+ Each group is the set of columns grouped at one rollup level; the empty
+ group ``()`` is the grand total. For example, groups ``[[a, b], [a], []]``
+ produce ``GROUPING SETS ((a, b), (a), ())``.
+
+ :param groups: one column list per rollup level
+ :return: a clause element suitable for ``select(...).group_by(...)``
+ """
+ return func.grouping_sets(*[tuple_(*group) for group in groups])
+
+
+def grouping_id_column(column: ColumnElement, label: str) -> ColumnElement:
+ """
+ Build a ``GROUPING(col) AS label`` marker column.
+
+ In a ``GROUPING SETS`` result, ``GROUPING(col)`` is ``0`` when ``col`` is
+ part of the row's grouping level and ``1`` when it has been rolled up
+ (aggregated away). Selecting one marker per groupby column lets the caller
+ attribute each returned row to its rollup level when splitting the single
+ result back into per-level results.
+
+ :param column: the groupby column to probe
+ :param label: the output label for the marker (see ``grouping_marker_label``)
+ :return: the labelled ``GROUPING(col)`` column
+ """
+ return func.grouping(column).label(label)
+
+
+def split_grouping_sets_result(
+ df: pd.DataFrame,
+ levels: Sequence[Sequence[str]],
+ groupby_columns: Sequence[str],
+) -> list[pd.DataFrame]:
+ """
+ Split a combined ``GROUPING SETS`` result into one DataFrame per rollup
+ level, the inverse of {@link grouping_sets_clause}.
Review Comment:
**Suggestion:** The docstring uses JavaDoc syntax (`{@link ...}`), which is not recognized by Python/Sphinx tooling and will render as literal text instead of a reference. Replace it with a valid Python/Sphinx reference format so generated docs and IDE help are correct. [typo]
<details>
<summary><b>Severity Level:</b> Minor 🧹</summary>
```mdx
⚠️ Docstring help shows raw {@link} token to developers.
⚠️ Generated API docs will not cross-link to grouping_sets_clause.
```
</details>
<details>
<summary><b>Steps of Reproduction ✅ </b></summary>
```mdx
1. Open `superset/common/grouping_sets.py` and locate `split_grouping_sets_result` at
lines 82-101 in the current PR version (verified via Read tool).
2. Inspect the function docstring at lines 87-95, specifically line 89 which reads `level,
the inverse of {@link grouping_sets_clause}.`.
3. Start a Python REPL, run `from superset.common.grouping_sets import
split_grouping_sets_result` and then `help(split_grouping_sets_result)` to view the
rendered docstring.
4. Observe that the text `{@link grouping_sets_clause}` appears literally in the help
output instead of being rendered as a cross-reference, since Python/Sphinx do not
interpret JavaDoc-style `{@link ...}` tokens.
```
</details>
[](https://app.codeant.ai/fix-in-ide?tool=cursor&prompt_id=b163608601ef40adb2a73499ef2760c8&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset) [](https://app.codeant.ai/fix-in-ide?tool=vscode-claude&prompt_id=b163608601ef40adb2a73499ef2760c8&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset)
*(Use Cmd/Ctrl + Click for best experience)*
<details>
<summary><b>Prompt for AI Agent 🤖 </b></summary>
```mdx
This is a comment left during a code review.
**Path:** superset/common/grouping_sets.py
**Line:** 89:89
**Comment:**
*Typo: The docstring uses JavaDoc syntax (`{@link ...}`), which is not recognized by Python/Sphinx tooling and will render as literal text instead of a reference. Replace it with a valid Python/Sphinx reference format so generated docs and IDE help are correct.
Validate the correctness of the flagged issue. If correct, How can I resolve this? If you propose a fix, implement it and please make it concise.
Once fix is implemented, also check other comments on the same PR, and ask user if the user wants to fix the rest of the comments as well. if said yes, then fetch all the comments validate the correctness and implement a minimal fix
```
</details>
<a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F41184&comment_hash=b375f6c7c3594f79cd7fbdc18efcef89ca9d9aaf1c0b6a9113c628b919fa2617&reaction=like'>👍</a> | <a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F41184&comment_hash=b375f6c7c3594f79cd7fbdc18efcef89ca9d9aaf1c0b6a9113c628b919fa2617&reaction=dislike'>👎</a>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(table/pivot-table): correct non-additive totals/subtotals via DB rollup [POC for SIP-216] [superset]
Posted by "codeant-ai-for-open-source[bot] (via GitHub)" <gi...@apache.org>.
codeant-ai-for-open-source[bot] commented on code in PR #41184:
URL: https://github.com/apache/superset/pull/41184#discussion_r3562190150
##########
superset-frontend/plugins/plugin-chart-pivot-table/src/react-pivottable/utilities.ts:
##########
@@ -383,6 +383,38 @@ const fmtNonString =
(x: string | number | null): string =>
typeof x === 'string' ? x : formatter(x as number);
+/*
+ * Passthrough "aggregator" for the multi-query pivot. Because the database
+ * already computed every rollup level (one query per level), each cell receives
Review Comment:
**Suggestion:** The new comment states that values come from “one query per level,” but this PR’s query path explicitly uses a single `GROUPING SETS` query when supported, so the comment now contradicts real behavior and can mislead future maintenance/debugging. [comment mismatch]
<details>
<summary><b>Severity Level:</b> Minor 🧹</summary>
```mdx
⚠️ Comment misstates query model used by PivotTable rollups.
⚠️ Future maintenance may mis-optimize based on incorrect assumptions.
```
</details>
<details>
<summary><b>Steps of Reproduction ✅ </b></summary>
```mdx
1. Inspect the non-additive query path in
`superset-frontend/plugins/plugin-chart-pivot-table/src/plugin/transformProps.ts:127-144`,
where the comment and code explicitly state that for non-additive metrics the pivot uses
“a single GROUPING SETS query” and then `splitGroupingSetsResult()` to split the combined
result into per-level frames.
2. Check the backend integration in `superset/superset/common/grouping_sets.py:20-27` and
`superset/superset/common/query_context_processor.py:259-279`, which documents that when a
datasource engine reports `supports_grouping_sets`, N per-rollup-level queries are
collapsed into a single native `GROUPING SETS` query, and only when native support is
absent does the processor fall back to “one query per rollup level” with concatenated
results.
3. Compare this architecture with the comment on `cellValue` in
`superset-frontend/plugins/plugin-chart-pivot-table/src/react-pivottable/utilities.ts:27-33`,
which describes it as a passthrough aggregator “for the multi-query pivot” and asserts
that “the database already computed every rollup level (one query per level),” treating
the multi-query fallback as the generic behavior.
4. Because `cellValue` is used by `PivotData` for all non-additive rollup results
regardless of whether the backend executed a single `GROUPING SETS` query or multiple
per-level queries (see `PivotData.constructor` at utilities.ts:53-103 and `transformProps`
at transformProps.ts:127-144), the “one query per level” wording in the comment is now
misleading and contradicts the supported single-query path, which can confuse future
maintainers about the actual query execution model.
```
</details>
[](https://app.codeant.ai/fix-in-ide?tool=cursor&prompt_id=451f89e004754b9d8b2edbc5f96e717d&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset) [](https://app.codeant.ai/fix-in-ide?tool=vscode-claude&prompt_id=451f89e004754b9d8b2edbc5f96e717d&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset)
*(Use Cmd/Ctrl + Click for best experience)*
<details>
<summary><b>Prompt for AI Agent 🤖 </b></summary>
```mdx
This is a comment left during a code review.
**Path:** superset-frontend/plugins/plugin-chart-pivot-table/src/react-pivottable/utilities.ts
**Line:** 388:388
**Comment:**
*Comment Mismatch: The new comment states that values come from “one query per level,” but this PR’s query path explicitly uses a single `GROUPING SETS` query when supported, so the comment now contradicts real behavior and can mislead future maintenance/debugging.
Validate the correctness of the flagged issue. If correct, How can I resolve this? If you propose a fix, implement it and please make it concise.
Once fix is implemented, also check other comments on the same PR, and ask user if the user wants to fix the rest of the comments as well. if said yes, then fetch all the comments validate the correctness and implement a minimal fix
```
</details>
<a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F41184&comment_hash=d5bcfae09a71fb8780b422161688367f40c419130a905972b5bd419d8c3a6c31&reaction=like'>👍</a> | <a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F41184&comment_hash=d5bcfae09a71fb8780b422161688367f40c419130a905972b5bd419d8c3a6c31&reaction=dislike'>👎</a>
##########
superset/db_engine_specs/presto.py:
##########
@@ -166,6 +166,7 @@ class PrestoBaseEngineSpec(BaseEngineSpec, metaclass=ABCMeta):
supports_dynamic_schema = True
supports_catalog = supports_dynamic_catalog = supports_cross_catalog_queries = True
+ supports_grouping_sets = True
Review Comment:
**Suggestion:** Setting this capability on `PrestoBaseEngineSpec` propagates to all subclasses in the inheritance chain (including `HiveEngineSpec`, `SparkEngineSpec`, and Databricks Hive), which can force the native GROUPING SETS path on engines that may not fully support the same `GROUPING()` marker semantics. That bypasses the fallback path and can cause runtime SQL failures for those engines; set the flag on `PrestoEngineSpec`/`TrinoEngineSpec` directly or explicitly override it to `False` on non-guaranteed descendants. [api mismatch]
<details>
<summary><b>Severity Level:</b> Major ⚠️</summary>
```mdx
❌ Pivot-table non-additive totals fail on Hive-like databases.
⚠️ Databricks Hive rollup queries may error on execution.
⚠️ SparkEngineSpec inherits flag; potential GROUPING SETS failures.
```
</details>
<details>
<summary><b>Steps of Reproduction ✅ </b></summary>
```mdx
1. Configure a database using Hive or a Hive-derived engine (e.g. Databricks Hive or
Spark), whose spec classes are `HiveEngineSpec` at `superset/db_engine_specs/hive.py:88`
(extends `PrestoEngineSpec`) and `DatabricksHiveEngineSpec` at
`superset/db_engine_specs/databricks.py:235` (extends `HiveEngineSpec`), both inheriting
`PrestoBaseEngineSpec` in `superset/db_engine_specs/presto.py:162-169` where
`supports_grouping_sets = True` is set.
2. Create a Pivot Table chart with a non-additive metric (such as a saved SQL metric, AVG,
or COUNT_DISTINCT) so the frontend `buildQuery` helper in
`superset-frontend/plugins/plugin-chart-pivot-table/src/plugin/buildQuery.ts:85-96`
populates `grouping_sets` on the single query and sends it to the backend `QueryContext`.
3. On the backend, the `QueryObject` with `grouping_sets` flows into
`QueryContextProcessor.get_query_result` in
`superset/common/query_context_processor.py:250-23`; `_supports_grouping_sets()` reads
`self._qc_datasource.db_engine_spec.supports_grouping_sets`, which is `True` for
`HiveEngineSpec`/`DatabricksHiveEngineSpec`/`SparkEngineSpec` via
`PrestoBaseEngineSpec.supports_grouping_sets` at `superset/db_engine_specs/presto.py:169`,
so the fallback `_grouping_sets_fallback()` is skipped and native `GROUPING SETS` SQL is
emitted.
4. The query builder in `superset/models/helpers.py:3564-3570` sees
`db_engine_spec.supports_grouping_sets` as `True`, appends `GROUPING()` marker columns via
`grouping_id_column` and issues a single `GROUPING SETS` query. On engines like
`HiveEngineSpec` (`superset/db_engine_specs/hive.py:88-134`) or `DatabricksHiveEngineSpec`
(`superset/db_engine_specs/databricks.py:235-252`) that do not fully support `GROUPING
SETS`/`GROUPING()` with the semantics assumed by `superset/common/grouping_sets.py:48-79`,
this SQL can fail at runtime (syntax errors or unsupported function), breaking Pivot Table
rollup queries instead of using the per-level fallback.
```
</details>
[](https://app.codeant.ai/fix-in-ide?tool=cursor&prompt_id=1348d6b183be400489ab101a09fa4028&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset) [](https://app.codeant.ai/fix-in-ide?tool=vscode-claude&prompt_id=1348d6b183be400489ab101a09fa4028&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset)
*(Use Cmd/Ctrl + Click for best experience)*
<details>
<summary><b>Prompt for AI Agent 🤖 </b></summary>
```mdx
This is a comment left during a code review.
**Path:** superset/db_engine_specs/presto.py
**Line:** 169:169
**Comment:**
*Api Mismatch: Setting this capability on `PrestoBaseEngineSpec` propagates to all subclasses in the inheritance chain (including `HiveEngineSpec`, `SparkEngineSpec`, and Databricks Hive), which can force the native GROUPING SETS path on engines that may not fully support the same `GROUPING()` marker semantics. That bypasses the fallback path and can cause runtime SQL failures for those engines; set the flag on `PrestoEngineSpec`/`TrinoEngineSpec` directly or explicitly override it to `False` on non-guaranteed descendants.
Validate the correctness of the flagged issue. If correct, How can I resolve this? If you propose a fix, implement it and please make it concise.
Once fix is implemented, also check other comments on the same PR, and ask user if the user wants to fix the rest of the comments as well. if said yes, then fetch all the comments validate the correctness and implement a minimal fix
```
</details>
<a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F41184&comment_hash=e110a706e9974575a22f844cbbf6bdeade3bd2df824f2b4f2011a4931934e67f&reaction=like'>👍</a> | <a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F41184&comment_hash=e110a706e9974575a22f844cbbf6bdeade3bd2df824f2b4f2011a4931934e67f&reaction=dislike'>👎</a>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(table/pivot-table): correct non-additive totals/subtotals via DB rollup [SIP-216] [superset]
Posted by "aminghadersohi (via GitHub)" <gi...@apache.org>.
aminghadersohi commented on PR #41184:
URL: https://github.com/apache/superset/pull/41184#issuecomment-5039967241
Following up on the earlier review: CI is currently red on this head (`9a30e5003e88`) in a way that looks like a real regression, not a flake.
`sharded-jest-tests (6)` fails 4 assertions in `plugin-chart-pivot-table/test/TableRenderer.test.tsx` (all about the `pvtRowLabelLast` class that marks the last data row before a totals row): `dataRows[dataRows.length - 1]` comes back `undefined`, and the "spanning" row-label header cell isn't found where expected.
That test file isn't part of this PR's own diff — it's an existing test that predates the rebase this branch picked up — and it was passing at the SHA this review was originally done against, so something in this PR's rewrite of the row/rollup logic is the likely cause rather than any of the small follow-up commits since (those only touch a control description, an unrelated table-chart test import, and a docstring).
My guess, not yet confirmed: `processRecord()`'s rewrite of how rows/subtotals get keyed and slotted (`react-pivottable/utilities.ts`) changed `rowKeys`/`visibleRowCount` in a way that breaks the last-row detection in `TableRenderers.tsx`'s `pvtRowLabelLast` logic. Worth a look before merge — this is separate from the total-value correctness reviewed earlier, which still looks solid.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(table/pivot-table): correct non-additive totals/subtotals via DB rollup [POC for SIP-216] [superset]
Posted by "codeant-ai-for-open-source[bot] (via GitHub)" <gi...@apache.org>.
codeant-ai-for-open-source[bot] commented on code in PR #41184:
URL: https://github.com/apache/superset/pull/41184#discussion_r3564126332
##########
superset/common/query_context_processor.py:
##########
@@ -252,9 +256,60 @@ def get_query_result(self, query_object: QueryObject) -> QueryResult:
This method delegates to the datasource's get_query_result method,
which handles query execution, normalization, time offsets, and
post-processing.
+
+ When the query requests rollup ``grouping_sets`` but the engine does not
+ support native ``GROUPING SETS``, fall back to one query per level and
+ concatenate the results with ``GROUPING()``-equivalent markers, so the
+ combined result matches the shape the native path produces (SIP.md,
+ phase 3b). Engines that support it run the single native query.
"""
+ if query_object.grouping_sets and not self._supports_grouping_sets():
+ return self._grouping_sets_fallback(query_object)
return self._qc_datasource.get_query_result(query_object)
+ def _supports_grouping_sets(self) -> bool:
+ engine_spec: BaseEngineSpec | None = getattr(
+ self._qc_datasource, "db_engine_spec", None
+ )
+ return bool(engine_spec and engine_spec.supports_grouping_sets)
+
+ def _grouping_sets_fallback(self, query_object: QueryObject) -> QueryResult:
+ """
+ Emulate a GROUPING SETS query on engines without native support: run one
+ query per rollup level and concatenate, tagging each level's rows with
+ the same per-column markers the native path emits.
+ """
+ levels: list[list[str]] = query_object.grouping_sets
+ # Use the same label derivation as the native path (physical column name
+ # or adhoc column label) so both column kinds are represented and each
+ # label maps back to its own column, in the same order as the source
+ # list.
+ all_labels = [get_column_name(col) for col in query_object.columns]
Review Comment:
**Suggestion:** Add an explicit type annotation for this newly introduced intermediate variable to keep the new logic fully type-hinted. [custom_rule]
**Severity Level:** Minor 🧹
<details>
<summary><b>Why it matters? ⭐ </b></summary>
This is newly added Python code and `all_labels` is a clearly annotatable local variable. It currently lacks an explicit type hint, so it matches the custom rule requiring type hints on new or modified Python code.
</details>
<details>
<summary><b>Rule source 📖 </b></summary>
.cursor/rules/dev-standard.mdc (line 28)
</details>
[](https://app.codeant.ai/fix-in-ide?tool=cursor&prompt_id=0ebd06c8475941e3aacbf0bdef87e9eb&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset) [](https://app.codeant.ai/fix-in-ide?tool=vscode-claude&prompt_id=0ebd06c8475941e3aacbf0bdef87e9eb&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset)
*(Use Cmd/Ctrl + Click for best experience)*
<details>
<summary><b>Prompt for AI Agent 🤖 </b></summary>
```mdx
This is a comment left during a code review.
**Path:** superset/common/query_context_processor.py
**Line:** 287:287
**Comment:**
*Custom Rule: Add an explicit type annotation for this newly introduced intermediate variable to keep the new logic fully type-hinted.
Validate the correctness of the flagged issue. If correct, How can I resolve this? If you propose a fix, implement it and please make it concise.
Once fix is implemented, also check other comments on the same PR, and ask user if the user wants to fix the rest of the comments as well. if said yes, then fetch all the comments validate the correctness and implement a minimal fix
```
</details>
<a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F41184&comment_hash=a1409c98891d0fd30df8a915e2d468d8b32fd3c509650e09c3974b837bf72e79&reaction=like'>👍</a> | <a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F41184&comment_hash=a1409c98891d0fd30df8a915e2d468d8b32fd3c509650e09c3974b837bf72e79&reaction=dislike'>👎</a>
##########
superset/common/grouping_sets.py:
##########
@@ -0,0 +1,116 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""
+SQL building blocks for the pivot-table non-additive totals optimization
+(SIP.md, phase 3b). When a datasource engine reports
+``supports_grouping_sets``, the N per-rollup-level queries can be collapsed into
+a single ``GROUPING SETS`` query: the database computes every level in one scan,
+and each returned row is attributed to its level via ``GROUPING()`` markers.
+
+These are the engine-agnostic SQL primitives. Wiring them into the query context
+(emitting one query and splitting the result back into per-level results) is the
+remaining integration; see SIP.md.
+"""
+
+from __future__ import annotations
+
+from collections.abc import Sequence
+
+import pandas as pd
+from sqlalchemy import func, tuple_
+from sqlalchemy.sql.elements import ColumnElement
+
+# Suffix for the per-column GROUPING() marker columns added to a GROUPING SETS
+# query. Chosen to be unlikely to collide with a real metric/column label.
+GROUPING_MARKER_SUFFIX = "__superset_grouping"
Review Comment:
**Suggestion:** Add an explicit type annotation to this module-level constant to comply with the required Python type hints rule. [custom_rule]
**Severity Level:** Minor 🧹
<details>
<summary><b>Why it matters? ⭐ </b></summary>
This new module-level constant is a relevant variable that can be annotated, but it is defined without a type hint. That matches the Python type-hints rule violation.
</details>
<details>
<summary><b>Rule source 📖 </b></summary>
.cursor/rules/dev-standard.mdc (line 28)
</details>
[](https://app.codeant.ai/fix-in-ide?tool=cursor&prompt_id=0d834547d92147acb201940ed0f75bf9&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset) [](https://app.codeant.ai/fix-in-ide?tool=vscode-claude&prompt_id=0d834547d92147acb201940ed0f75bf9&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset)
*(Use Cmd/Ctrl + Click for best experience)*
<details>
<summary><b>Prompt for AI Agent 🤖 </b></summary>
```mdx
This is a comment left during a code review.
**Path:** superset/common/grouping_sets.py
**Line:** 38:40
**Comment:**
*Custom Rule: Add an explicit type annotation to this module-level constant to comply with the required Python type hints rule.
Validate the correctness of the flagged issue. If correct, How can I resolve this? If you propose a fix, implement it and please make it concise.
Once fix is implemented, also check other comments on the same PR, and ask user if the user wants to fix the rest of the comments as well. if said yes, then fetch all the comments validate the correctness and implement a minimal fix
```
</details>
<a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F41184&comment_hash=f97816643f9864c3ec0896351478c2b85c84adf03e77915009a8676963c59299&reaction=like'>👍</a> | <a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F41184&comment_hash=f97816643f9864c3ec0896351478c2b85c84adf03e77915009a8676963c59299&reaction=dislike'>👎</a>
##########
superset/db_engine_specs/hive.py:
##########
@@ -96,6 +96,11 @@ class HiveEngineSpec(PrestoEngineSpec):
supports_dynamic_schema = True
supports_cross_catalog_queries = False
+ # Explicitly opt out (overriding the inherited PrestoEngineSpec value):
+ # Hive/Spark's GROUPING SETS + GROUPING() marker semantics have not been
+ # verified against this query pattern, so fall back to one query per
+ # rollup level instead of assuming native support.
+ supports_grouping_sets = False
Review Comment:
**Suggestion:** Add an explicit type annotation to this newly introduced class-level boolean setting. [custom_rule]
**Severity Level:** Minor 🧹
<details>
<summary><b>Why it matters? ⭐ </b></summary>
This is a newly added class-level boolean assignment in Python that can be annotated as `supports_grouping_sets: bool = False`, so it matches the rule about missing type hints on annotatable variables.
</details>
<details>
<summary><b>Rule source 📖 </b></summary>
.cursor/rules/dev-standard.mdc (line 28)
</details>
[](https://app.codeant.ai/fix-in-ide?tool=cursor&prompt_id=ca28652cbd2e400fba8d1e9bbdb5001b&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset) [](https://app.codeant.ai/fix-in-ide?tool=vscode-claude&prompt_id=ca28652cbd2e400fba8d1e9bbdb5001b&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset)
*(Use Cmd/Ctrl + Click for best experience)*
<details>
<summary><b>Prompt for AI Agent 🤖 </b></summary>
```mdx
This is a comment left during a code review.
**Path:** superset/db_engine_specs/hive.py
**Line:** 103:103
**Comment:**
*Custom Rule: Add an explicit type annotation to this newly introduced class-level boolean setting.
Validate the correctness of the flagged issue. If correct, How can I resolve this? If you propose a fix, implement it and please make it concise.
Once fix is implemented, also check other comments on the same PR, and ask user if the user wants to fix the rest of the comments as well. if said yes, then fetch all the comments validate the correctness and implement a minimal fix
```
</details>
<a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F41184&comment_hash=0955dc89e0fed2ffe3b63f9193b15e7a72614acb6000c66d862f6b3601021b0b&reaction=like'>👍</a> | <a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F41184&comment_hash=0955dc89e0fed2ffe3b63f9193b15e7a72614acb6000c66d862f6b3601021b0b&reaction=dislike'>👎</a>
##########
superset-frontend/plugins/plugin-chart-pivot-table/test/plugin/buildQuery.test.ts:
##########
@@ -56,9 +55,41 @@ const formData: PivotTableQueryFormData = {
currencyFormat: { symbol: 'USD', symbolPosition: 'prefix' },
};
-test('should build groupby with series in form data', () => {
+test('additive metrics use the fast-path: a single full-detail query', () => {
+ const { queries } = buildQuery({
+ ...formData,
+ metrics: [
+ {
+ expressionType: 'SIMPLE',
+ aggregate: 'SUM',
+ column: { column_name: 'num' },
+ label: 'sum_num',
+ },
+ ] as any,
+ });
Review Comment:
**Suggestion:** Replace the `any` cast on the `metrics` value with the correct concrete metric array type expected by `PivotTableQueryFormData`. [custom_rule]
**Severity Level:** Minor 🧹
<details>
<summary><b>Why it matters? ⭐ </b></summary>
The changed TypeScript test code introduces an `any` cast for the `metrics` value. This directly violates the no-any-types rule, which requires a concrete type instead of `any`.
</details>
<details>
<summary><b>Rule source 📖 </b></summary>
.cursor/rules/dev-standard.mdc (line 16)
</details>
[](https://app.codeant.ai/fix-in-ide?tool=cursor&prompt_id=298b6f437a494df782f4109ed1b94118&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset) [](https://app.codeant.ai/fix-in-ide?tool=vscode-claude&prompt_id=298b6f437a494df782f4109ed1b94118&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset)
*(Use Cmd/Ctrl + Click for best experience)*
<details>
<summary><b>Prompt for AI Agent 🤖 </b></summary>
```mdx
This is a comment left during a code review.
**Path:** superset-frontend/plugins/plugin-chart-pivot-table/test/plugin/buildQuery.test.ts
**Line:** 58:69
**Comment:**
*Custom Rule: Replace the `any` cast on the `metrics` value with the correct concrete metric array type expected by `PivotTableQueryFormData`.
Validate the correctness of the flagged issue. If correct, How can I resolve this? If you propose a fix, implement it and please make it concise.
Once fix is implemented, also check other comments on the same PR, and ask user if the user wants to fix the rest of the comments as well. if said yes, then fetch all the comments validate the correctness and implement a minimal fix
```
</details>
<a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F41184&comment_hash=fe07a5c9946e7962bf62533545649419734ab43a1317a58912ac765d6d754632&reaction=like'>👍</a> | <a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F41184&comment_hash=fe07a5c9946e7962bf62533545649419734ab43a1317a58912ac765d6d754632&reaction=dislike'>👎</a>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(table/pivot-table): correct non-additive totals/subtotals via DB rollup [POC for SIP-216] [superset]
Posted by "bito-code-review[bot] (via GitHub)" <gi...@apache.org>.
bito-code-review[bot] commented on code in PR #41184:
URL: https://github.com/apache/superset/pull/41184#discussion_r3557748333
##########
superset-frontend/plugins/plugin-chart-pivot-table/test/react-pivottable/tableRenders.test.tsx:
##########
@@ -103,35 +150,41 @@ test('TableRenderer renders row headers from pivot data', () => {
});
test('TableRenderer renders aggregated cell values', () => {
- const props = buildDefaultProps();
+ const props = buildDefaultProps({
+ data: TAGGED_COUNT_DATA,
+ vals: ['value'],
+ });
renderWithTheme(<TableRenderer {...props} />);
- // With "Count" aggregator, each cell (row x col intersection) should
- // contain "1" because each combination appears exactly once.
+ // Each leaf cell (row x col intersection) holds the DB-computed value "1".
const cells = screen.getAllByRole('gridcell');
const cellTexts = cells.map(cell => cell.textContent);
- // There should be cell values of "1" for each of the four intersections
- // (blue+circle, blue+square, red+circle, red+square).
- const onesCount = cellTexts.filter(text => text === '1').length;
+ // There should be a "1" leaf cell for each of the four intersections
+ // (blue+circle, blue+square, red+circle, red+square). The default formatter
+ // renders with two decimals in this test (production applies the metric's
+ // own format).
+ const onesCount = cellTexts.filter(text => text === '1.00').length;
Review Comment:
<!-- Bito Reply -->
The suggestion is correct. The test uses `as any` to cast the `aggregatorTemplates.count()` result, which causes the formatter to default to `usFmtInt` (0 decimal places) instead of the expected 2-decimal format. To resolve this, you should either apply the correct formatter in the test or adjust the assertion to match the actual output.
**superset-frontend/plugins/plugin-chart-pivot-table/test/react-pivottable/tableRenders.test.tsx**
```
// There should be a "1" leaf cell for each of the four intersections
// (blue+circle, blue+square, red+circle, red+square). The default formatter
// renders with two decimals in this test (production applies the metric's
// own format).
const onesCount = cellTexts.filter(text => text === '1.00').length;
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(table/pivot-table): correct non-additive totals/subtotals via DB rollup [POC for SIP-216] [superset]
Posted by "rusackas (via GitHub)" <gi...@apache.org>.
rusackas commented on code in PR #41184:
URL: https://github.com/apache/superset/pull/41184#discussion_r3564072460
##########
superset-frontend/plugins/plugin-chart-pivot-table/src/PivotTableChart.tsx:
##########
@@ -341,22 +296,43 @@ export default function PivotTableChart(props: PivotTableProps) {
const unpivotedData = useMemo(
() =>
- data.reduce(
- (acc: Record<string, any>[], record: Record<string, any>) => [
- ...acc,
- ...metricNames
+ // `data` is now one entry per rollup level. Tag every record with the
+ // row/column dimension labels of the level that produced it (mirroring
+ // the METRIC_KEY injection used for the full rows/cols below) so
+ // PivotData can slot each pre-computed value without re-aggregating.
+ // buildGroupbyCombinations already applied transposePivot, so the level's
+ // groupby is display-oriented and is not transposed again here.
+ data.flatMap((query: QueryData) => {
+ let levelRows = query.groupby.rows.map(getColumnLabel);
+ let levelCols = query.groupby.columns.map(getColumnLabel);
+ if (metricsLayout === MetricsLayoutEnum.ROWS) {
+ levelRows = combineMetric
+ ? [...levelRows, METRIC_KEY]
+ : [METRIC_KEY, ...levelRows];
+ } else {
+ levelCols = combineMetric
+ ? [...levelCols, METRIC_KEY]
+ : [METRIC_KEY, ...levelCols];
+ }
+ return query.data.flatMap((record: Record<string, any>) =>
+ metricNames
.map((name: string) => ({
...record,
[METRIC_KEY]: name,
value: record[name],
// Mark currency column for per-cell currency detection in aggregators
__currencyColumn: currencyCodeColumn,
+ // The level this record belongs to (used by PivotData placement).
+ rows: levelRows,
+ columns: levelCols,
Review Comment:
Good catch — renamed the rollup tags to `__rows`/`__columns` (matching the existing `__metricKey`/`__currencyColumn` convention) so they can't collide with a real dataset column named `rows`/`columns`.
##########
superset-frontend/plugins/plugin-chart-pivot-table/src/react-pivottable/utilities.ts:
##########
@@ -383,6 +383,38 @@ const fmtNonString =
(x: string | number | null): string =>
typeof x === 'string' ? x : formatter(x as number);
+/*
+ * Passthrough "aggregator" for the multi-query pivot. Because the database
+ * already computed every rollup level (one query per level), each cell receives
Review Comment:
Fixed the comment to describe both paths (single GROUPING SETS query or per-level fallback) instead of only the fallback.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(table/pivot-table): correct non-additive totals/subtotals via DB rollup [POC for SIP-216] [superset]
Posted by "bito-code-review[bot] (via GitHub)" <gi...@apache.org>.
bito-code-review[bot] commented on code in PR #41184:
URL: https://github.com/apache/superset/pull/41184#discussion_r3565629800
##########
superset-frontend/plugins/plugin-chart-pivot-table/src/plugin/buildQuery.ts:
##########
@@ -54,6 +59,39 @@ export default function buildQuery(formData: PivotTableQueryFormData) {
}
return col;
});
+}
+
+export default function buildQuery(formData: PivotTableQueryFormData) {
+ const { extra_form_data } = formData;
+ const time_grain_sqla =
+ extra_form_data?.time_grain_sqla || formData.time_grain_sqla;
Review Comment:
<!-- Bito Reply -->
The suggestion to forward `time_grain_sqla` to `extras.time_grain_sqla` is correct. While `extra_form_data` is extracted, it must be explicitly included in the `extras` object returned by `buildQuery` for the database engine to receive and apply the dashboard-level override. The existing test confirms this requirement, and the current implementation is missing this assignment.
**superset-frontend/plugins/plugin-chart-pivot-table/src/plugin/buildQuery.ts**
```
const { extra_form_data } = formData;
const time_grain_sqla =
extra_form_data?.time_grain_sqla || formData.time_grain_sqla;
// ...
return {
...query,
extras: {
...query.extras,
time_grain_sqla,
},
};
```
##########
superset-frontend/plugins/plugin-chart-pivot-table/test/react-pivottable/tableRenders.test.tsx:
##########
@@ -149,18 +202,83 @@ test('TableRenderer renders col totals row when colTotals is enabled', () => {
test('TableRenderer renders grand total when both totals are enabled', () => {
const props = buildDefaultProps({
+ data: TAGGED_COUNT_DATA,
+ vals: ['value'],
tableOptions: { rowTotals: true, colTotals: true },
});
renderWithTheme(<TableRenderer {...props} />);
- // The grand total cell should show "4" (total record count).
+ // The grand total cell shows the DB-computed "4" (total record count).
const grandTotalCells = screen
.getAllByRole('gridcell')
.filter(cell => cell.classList.contains('pvtGrandTotal'));
expect(grandTotalCells.length).toBe(1);
expect(grandTotalCells[0]).toHaveTextContent('4');
Review Comment:
<!-- Bito Reply -->
The assertion `expect(grandTotalCells[0]).toHaveTextContent('4')` in the test file `superset-frontend/plugins/plugin-chart-pivot-table/test/react-pivottable/tableRenders.test.tsx` does not perform a substring match by default. The `toHaveTextContent` matcher in Jest/Testing Library checks if the element's text content contains the provided string, but it is sensitive to the exact string representation. If the formatter renders the value as '4.00' instead of '4', the assertion will fail because '4' is not equal to '4.00'.
**superset-frontend/plugins/plugin-chart-pivot-table/test/react-pivottable/tableRenders.test.tsx**
```
// The grand total cell shows the DB-computed "4" (total record count).
const grandTotalCells = screen
.getAllByRole('gridcell')
.filter(cell => cell.classList.contains('pvtGrandTotal'));
expect(grandTotalCells.length).toBe(1);
expect(grandTotalCells[0]).toHaveTextContent('4');
```
##########
superset-frontend/plugins/plugin-chart-pivot-table/src/react-pivottable/utilities.ts:
##########
@@ -1088,76 +1117,108 @@ class PivotData {
}
processRecord(record: PivotRecord): void {
- // this code is called in a tight loop
+ // this code is called in a tight loop.
+ // Each record is tagged (in PivotTableChart) with `__rows`/`__columns`:
+ // the dimension labels of the rollup level that produced it. The database
+ // has already aggregated that level, so we place the value into exactly
+ // one slot and store it verbatim (no re-aggregation). A key shorter than
+ // the full dimension list identifies a subtotal level. Records without
+ // tags (e.g. direct unit-test construction) fall back to the full
+ // dimension lists, i.e. they are treated as leaf rows. The `__` prefix
+ // keeps these rollup tags from colliding with a real dataset column
+ // named `rows`/`columns`.
+ const recordRows = (record.__rows as unknown as string[]) ?? null;
+ const recordColumns = (record.__columns as unknown as string[]) ?? null;
+ const levelRows = recordRows ?? (this.props.rows as string[]);
+ const levelColumns = recordColumns ?? (this.props.cols as string[]);
+
const colKey: string[] = [];
const rowKey: string[] = [];
- (this.props.cols as string[]).forEach((col: string) => {
+ levelColumns.forEach((col: string) => {
colKey.push(col in record ? String(record[col]) : 'null');
});
- (this.props.rows as string[]).forEach((row: string) => {
+ levelRows.forEach((row: string) => {
rowKey.push(row in record ? String(record[row]) : 'null');
});
- this.allTotal.push(record);
-
- const rowStart = this.subtotals.rowEnabled ? 1 : Math.max(1, rowKey.length);
- const colStart = this.subtotals.colEnabled ? 1 : Math.max(1, colKey.length);
-
- let isRowSubtotal;
- let isColSubtotal;
- for (let ri = rowStart; ri <= rowKey.length; ri += 1) {
- isRowSubtotal = ri < rowKey.length;
- const fRowKey = rowKey.slice(0, ri);
- const flatRowKey = flatKey(fRowKey);
- if (!this.rowTotals[flatRowKey]) {
- this.rowKeys.push(fRowKey);
- this.rowTotals[flatRowKey] = this.getFormattedAggregator(
- record,
- rowKey,
- )(this, fRowKey, []);
+ const flatRowKey = flatKey(rowKey);
+ const flatColKey = flatKey(colKey);
+
+ const isColSubtotal = colKey.length < (this.props.cols as string[]).length;
+ const isRowSubtotal = rowKey.length < (this.props.rows as string[]).length;
+
+ // Register/create the column-total slot the first time this colKey is seen.
+ if (colKey.length > 0 && !this.colTotals[flatColKey]) {
+ if (!isColSubtotal || this.subtotals.colEnabled) {
+ this.colKeys.push(colKey);
}
- this.rowTotals[flatRowKey].push(record);
- this.rowTotals[flatRowKey].isSubtotal = isRowSubtotal;
+ this.colTotals[flatColKey] = this.getFormattedAggregator(record, colKey)(
+ this,
+ [],
+ colKey,
+ );
}
-
- for (let ci = colStart; ci <= colKey.length; ci += 1) {
- isColSubtotal = ci < colKey.length;
- const fColKey = colKey.slice(0, ci);
- const flatColKey = flatKey(fColKey);
- if (!this.colTotals[flatColKey]) {
- this.colKeys.push(fColKey);
- this.colTotals[flatColKey] = this.getFormattedAggregator(
- record,
+ // Register/create the row-total slot the first time this rowKey is seen.
+ if (rowKey.length > 0 && !this.rowTotals[flatRowKey]) {
+ if (!isRowSubtotal || this.subtotals.rowEnabled) {
+ this.rowKeys.push(rowKey);
+ }
+ this.rowTotals[flatRowKey] = this.getFormattedAggregator(record, rowKey)(
+ this,
+ rowKey,
+ [],
+ );
+ }
+ // Create the body-cell slot.
+ if (rowKey.length > 0 && colKey.length > 0) {
+ if (!this.tree[flatRowKey]) {
+ this.tree[flatRowKey] = {};
+ }
+ if (!this.tree[flatRowKey][flatColKey]) {
+ this.tree[flatRowKey][flatColKey] = this.getFormattedAggregator(record)(
+ this,
+ rowKey,
colKey,
- )(this, [], fColKey);
+ );
}
+ }
+
+ // Place the value in exactly one slot, determined by the level.
+ if (rowKey.length === 0 && colKey.length === 0) {
+ this.allTotal.push(record);
+ } else if (rowKey.length === 0) {
this.colTotals[flatColKey].push(record);
this.colTotals[flatColKey].isSubtotal = isColSubtotal;
+ } else if (colKey.length === 0) {
+ this.rowTotals[flatRowKey].push(record);
+ this.rowTotals[flatRowKey].isSubtotal = isRowSubtotal;
+ } else {
+ this.tree[flatRowKey][flatColKey].push(record);
+ this.tree[flatRowKey][flatColKey].isRowSubtotal = isRowSubtotal;
+ this.tree[flatRowKey][flatColKey].isColSubtotal = isColSubtotal;
+ this.tree[flatRowKey][flatColKey].isSubtotal =
+ isRowSubtotal || isColSubtotal;
}
- // And now fill in for all the sub-cells.
- for (let ri = rowStart; ri <= rowKey.length; ri += 1) {
- isRowSubtotal = ri < rowKey.length;
- const fRowKey = rowKey.slice(0, ri);
- const flatRowKey = flatKey(fRowKey);
- if (!this.tree[flatRowKey]) {
- this.tree[flatRowKey] = {};
+ // Metric-collapse totals. The metric is a pseudo-dimension always present on
+ // one axis, so no rollup level produces an empty key on that axis -- which
+ // would leave the opposite "Total" axis and the grand-total corner empty.
+ // When a record's axis holds only the metric (no real dims there), its value
+ // is also the collapsed total for that axis, so mirror it into rowTotals /
+ // colTotals / allTotal. (For a single metric this equals the metric column;
+ // for multiple metrics it is the last metric -- a cross-metric total is not
+ // well defined and is left as future work.)
+ const metricKey = record.__metricKey as unknown as string | undefined;
+ if (metricKey) {
+ const realColCount = levelColumns.filter(c => c !== metricKey).length;
+ const realRowCount = levelRows.filter(r => r !== metricKey).length;
+ if (levelColumns.includes(metricKey) && realColCount === 0) {
+ if (rowKey.length === 0) this.allTotal.push(record);
+ else this.rowTotals[flatRowKey]?.push(record);
}
- for (let ci = colStart; ci <= colKey.length; ci += 1) {
- isColSubtotal = ci < colKey.length;
- const fColKey = colKey.slice(0, ci);
- const flatColKey = flatKey(fColKey);
- if (!this.tree[flatRowKey][flatColKey]) {
- this.tree[flatRowKey][flatColKey] = this.getFormattedAggregator(
- record,
- )(this, fRowKey, fColKey);
- }
- this.tree[flatRowKey][flatColKey].push(record);
-
- this.tree[flatRowKey][flatColKey].isRowSubtotal = isRowSubtotal;
- this.tree[flatRowKey][flatColKey].isColSubtotal = isColSubtotal;
- this.tree[flatRowKey][flatColKey].isSubtotal =
- isRowSubtotal || isColSubtotal;
+ if (levelRows.includes(metricKey) && realRowCount === 0) {
+ if (colKey.length === 0) this.allTotal.push(record);
+ else this.colTotals[flatColKey]?.push(record);
}
Review Comment:
<!-- Bito Reply -->
The suggestion to add `this.subtotals.rowEnabled` and `this.subtotals.colEnabled` checks to the metric-collapse logic is valid. This ensures that the metric-collapse totals are only mirrored when the corresponding subtotals are enabled, maintaining consistency with the guard pattern used elsewhere in the `processRecord` method.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(table/pivot-table): correct non-additive totals/subtotals via DB rollup [POC for SIP-216] [superset]
Posted by "rusackas (via GitHub)" <gi...@apache.org>.
rusackas commented on code in PR #41184:
URL: https://github.com/apache/superset/pull/41184#discussion_r3565630714
##########
superset-frontend/plugins/plugin-chart-pivot-table/src/react-pivottable/TableRenderers.tsx:
##########
@@ -1448,9 +1437,7 @@ export function TableRenderer(props: TableRendererProps) {
true,
)}
>
- {t('Total (%(aggregatorName)s)', {
- aggregatorName: t(aggregatorName),
- })}
+ {t('Total')}
Review Comment:
Same as the other `aggregatorName` threads on this file, this is intentional per UPDATING.md since the Aggregation function control is gone.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(table/pivot-table): correct non-additive totals/subtotals via DB rollup [SIP-216] [superset]
Posted by "bito-code-review[bot] (via GitHub)" <gi...@apache.org>.
bito-code-review[bot] commented on PR #41184:
URL: https://github.com/apache/superset/pull/41184#issuecomment-5038905546
AI Code Review is in progress (usually takes 3 to 15 minutes unless it's a very large PR).
<details>
<summary>
Bito Usage Guide
</summary>
**Commands**
Type the following command in the pull request comment and save the comment.
- `/review` - Manually triggers a full AI review.
- `/pause` - Pauses automatic reviews on this pull request.
- `/resume` - Resumes automatic reviews.
- `/resolve` - Marks all Bito-posted review comments as resolved.
- `/abort` - Cancels all in-progress reviews.
Refer to the <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/available-commands" target="_blank" rel="noopener noreferrer">documentation</a> for additional commands.
**Configuration**
This repository uses `Superset` You can customize the agent settings <a href="https://alpha.bito.ai/home/ai-agents/code-review-agent" target="_blank" rel="noopener noreferrer">here</a> or contact your Bito workspace admin at evan@preset.io.
**Documentation & Help**
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/getting-started/install-run-using-bito-cloud/create-or-customize-an-agent-instance" target="_blank" rel="noopener noreferrer">Customize agent settings</a>
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/implementing-custom-code-review-rules" target="_blank" rel="noopener noreferrer">Review rules</a>
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent" target="_blank" rel="noopener noreferrer">General documentation</a>
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/faqs" target="_blank" rel="noopener noreferrer">FAQ</a>
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] [SIP] Correct totals/subtotals for non-additive metrics (Table + Pivot) [superset]
Posted by "netlify[bot] (via GitHub)" <gi...@apache.org>.
netlify[bot] commented on PR #41184:
URL: https://github.com/apache/superset/pull/41184#issuecomment-4744180635
### <span aria-hidden="true">✅</span> Deploy Preview for *superset-docs-preview* ready!
| Name | Link |
|:-:|------------------------|
|<span aria-hidden="true">🔨</span> Latest commit | b685383445f6878998a08a9eb80fc4dc76881bdd |
|<span aria-hidden="true">🔍</span> Latest deploy log | https://app.netlify.com/projects/superset-docs-preview/deploys/6a342007f001a300086c1d2c |
|<span aria-hidden="true">😎</span> Deploy Preview | [https://deploy-preview-41184--superset-docs-preview.netlify.app](https://deploy-preview-41184--superset-docs-preview.netlify.app) |
|<span aria-hidden="true">📱</span> Preview on mobile | <details><summary> Toggle QR Code... </summary><br /><br /><br /><br />_Use your smartphone camera to open QR code link._</details> |
|<span aria-hidden="true">🤖</span> Make changes | [Run an agent on this branch](https://app.netlify.com/projects/superset-docs-preview/agent-runs#DcpBCoAgEADAv-w5YbWw6Ac9Y3ONBHNFJYjo73mbw7yQi1y5wQowwF4oubO7hqySJEXMoYXbqyaNYu2FfY7ybNyTpXEyiPOBqGlExMU6zcbB9wM) |
---
<!-- [superset-docs-preview Preview](https://deploy-preview-41184--superset-docs-preview.netlify.app) -->
_To edit notification comments on pull requests, go to your [Netlify project configuration](https://app.netlify.com/projects/superset-docs-preview/configuration/notifications#deploy-notifications)._
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(table/pivot-table): correct non-additive totals/subtotals via DB rollup [SIP-216] [superset]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #41184:
URL: https://github.com/apache/superset/pull/41184#issuecomment-5008550558
🎪 [Showtime](https://github.com/mistercrunch/superset-showtime) is building environment on [GHA](https://github.com/apache/superset/actions/runs/29621220321) for [67b8a00](https://github.com/apache/superset/commit/67b8a00)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(table/pivot-table): correct non-additive totals/subtotals via DB rollup [POC for SIP-216] [superset]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #41184:
URL: https://github.com/apache/superset/pull/41184#issuecomment-4988624923
🎪 [Showtime](https://github.com/mistercrunch/superset-showtime) is building environment on [GHA](https://github.com/apache/superset/actions/runs/29474785437) for [67b8a00](https://github.com/apache/superset/commit/67b8a00)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(table/pivot-table): correct non-additive totals/subtotals via DB rollup [SIP-216] [superset]
Posted by "rusackas (via GitHub)" <gi...@apache.org>.
rusackas commented on PR #41184:
URL: https://github.com/apache/superset/pull/41184#issuecomment-5039017513
Thanks for going through the injection surface and the additive/non-additive split, @aminghadersohi. Added a comment in `_grouping_sets_fallback` noting the per-level query fan-out for engines without native `GROUPING SETS` support, per your POC-scope note.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(table/pivot-table): correct non-additive totals/subtotals via DB rollup [SIP-216] [superset]
Posted by "rusackas (via GitHub)" <gi...@apache.org>.
rusackas commented on PR #41184:
URL: https://github.com/apache/superset/pull/41184#issuecomment-5040607431
Thanks for catching this, @aminghadersohi. Turned out `splitGroupingSetsResult` was dropping every row when the frame had no `GROUPING()` marker columns (any query that never hit the grouping-sets extra), so it now falls back to attributing them to the leaf level. `sharded-jest-tests (6)` is green on the current head.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(table/pivot-table): correct non-additive totals/subtotals via DB rollup [SIP-216] [superset]
Posted by "rusackas (via GitHub)" <gi...@apache.org>.
rusackas commented on code in PR #41184:
URL: https://github.com/apache/superset/pull/41184#discussion_r3624410628
##########
superset/models/helpers.py:
##########
@@ -3962,7 +3996,10 @@ def get_sqla_query( # pylint: disable=too-many-arguments,too-many-locals,too-ma
direction = sa.asc if ascending else sa.desc
qry = qry.order_by(direction(col))
- if row_limit:
+ # A GROUPING SETS query computes a bounded set of rollup levels; applying
+ # a row LIMIT is both unnecessary and unparseable by some SQL parsers
+ # (LIMIT after a GROUPING SETS GROUP BY), so skip it for that path.
+ if row_limit and not use_grouping_sets:
qry = qry.limit(row_limit)
if row_offset:
qry = qry.offset(row_offset)
Review Comment:
This is intentional. GROUPING SETS computes a bounded set of rollup levels, so limiting it risks cutting off subtotal/grand-total rows, and some engines can't even parse LIMIT after a GROUPING SETS GROUP BY. Added a note to the Cell limit control's description so users know it doesn't apply to non-additive rollups.
##########
superset/common/query_context_processor.py:
##########
@@ -252,9 +256,78 @@ def get_query_result(self, query_object: QueryObject) -> QueryResult:
This method delegates to the datasource's get_query_result method,
which handles query execution, normalization, time offsets, and
post-processing.
+
+ When the query requests rollup ``grouping_sets`` but the engine does not
+ support native ``GROUPING SETS``, fall back to one query per level and
+ concatenate the results with ``GROUPING()``-equivalent markers, so the
+ combined result matches the shape the native path produces (SIP.md,
+ phase 3b). Engines that support it run the single native query.
"""
+ if query_object.grouping_sets and not self._supports_grouping_sets():
+ return self._grouping_sets_fallback(query_object)
return self._qc_datasource.get_query_result(query_object)
+ def _supports_grouping_sets(self) -> bool:
+ engine_spec: BaseEngineSpec | None = getattr(
+ self._qc_datasource, "db_engine_spec", None
+ )
+ return bool(engine_spec and engine_spec.supports_grouping_sets)
+
+ def _grouping_sets_fallback(self, query_object: QueryObject) -> QueryResult:
+ """
+ Emulate a GROUPING SETS query on engines without native support: run one
+ query per rollup level and concatenate, tagging each level's rows with
+ the same per-column markers the native path emits.
+ """
+ levels: list[list[str]] = query_object.grouping_sets
+ # Use the same label derivation as the native path (physical column name
+ # or adhoc column label) so both column kinds are represented and each
+ # label maps back to its own column, in the same order as the source
+ # list.
+ all_labels: list[str] = [get_column_name(col) for col in query_object.columns]
+ label_to_column: dict[str, Column] = dict(
+ zip(all_labels, query_object.columns, strict=True)
+ )
+
+ frames: list[pd.DataFrame] = []
+ result: QueryResult | None = None
+ for level in levels:
+ level_labels: set[str] = set(level)
+ sub_query = copy.copy(query_object)
+ sub_query.grouping_sets = []
+ sub_query.columns = [
+ label_to_column[label] for label in all_labels if label in level_labels
+ ]
+ # A GROUPING SETS query computes a bounded set of rollup levels, so
+ # the native path never applies row_limit to it (see the
+ # `use_grouping_sets` check in models/helpers.py). Match that here:
+ # limiting each level's fallback sub-query independently would
+ # truncate subtotal/grand-total rows and diverge from the native
+ # result shape. The native path applies `row_offset` exactly once,
+ # to the combined multi-level result (see the unconditional
+ # `qry.offset()` call in models/helpers.py). Applying the same
+ # offset to each per-level sub-query independently would apply it
+ # once per level instead of once overall, and can silently drop
+ # low-row-count levels (e.g. the single grand-total row) entirely.
+ # Zero it here and apply it once after concatenation instead.
+ sub_query.row_limit = None
+ sub_query.row_offset = 0
+ result = self._qc_datasource.get_query_result(sub_query)
Review Comment:
Matches the native path on purpose, so the fallback behaves the same as native GROUPING SETS regardless of engine support, rather than silently truncating rollup levels on engines without it. Same reasoning as the helpers.py thread. Documented the limitation on the Cell limit control instead of enforcing it here.
##########
superset-frontend/plugins/plugin-chart-pivot-table/test/plugin/buildQuery.test.ts:
##########
@@ -17,18 +17,27 @@
* under the License.
*/
-import { TimeGranularity } from '@superset-ui/core';
+import {
+ QueryFormColumn,
+ QueryFormMetric,
+ QueryObject,
+ TimeGranularity,
+} from '@superset-ui/core';
import buildQuery from '../../src/plugin/buildQuery';
import { PivotTableQueryFormData } from '../../src/types';
+// buildQuery attaches `grouping_sets` (one entry per rollup level) to the
+// query object; it is not part of the base QueryObject type, so narrow to a
+// local type instead of casting through `any`.
+type GroupingSetsQuery = QueryObject & { grouping_sets: QueryFormColumn[][] };
Review Comment:
Good catch, `grouping_sets` is column labels, not `QueryFormColumn[][]`. Fixed the test's type to `string[][]` and dropped the unused import.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(table/pivot-table): correct non-additive totals/subtotals via DB rollup [POC for SIP-216] [superset]
Posted by "codeant-ai-for-open-source[bot] (via GitHub)" <gi...@apache.org>.
codeant-ai-for-open-source[bot] commented on code in PR #41184:
URL: https://github.com/apache/superset/pull/41184#discussion_r3557744988
##########
superset/charts/schemas.py:
##########
@@ -1358,6 +1358,17 @@ class Meta: # pylint: disable=too-few-public-methods
load_default=False,
allow_none=True,
)
+ grouping_sets = fields.List(
Review Comment:
**Suggestion:** Add an explicit type annotation to this newly introduced schema field attribute so the new code complies with the repository's type-hint requirement. [custom_rule]
**Severity Level:** Minor 🧹
<details>
<summary><b>Why it matters? ⭐ </b></summary>
This is newly added Python code and the class attribute is introduced without any type annotation, which matches the repository rule requiring type hints on annotatable variables.
</details>
<details>
<summary><b>Rule source 📖 </b></summary>
.cursor/rules/dev-standard.mdc (line 28)
</details>
[](https://app.codeant.ai/fix-in-ide?tool=cursor&prompt_id=dad7a907f96844bcb969d01667475951&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset) [](https://app.codeant.ai/fix-in-ide?tool=vscode-claude&prompt_id=dad7a907f96844bcb969d01667475951&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset)
*(Use Cmd/Ctrl + Click for best experience)*
<details>
<summary><b>Prompt for AI Agent 🤖 </b></summary>
```mdx
This is a comment left during a code review.
**Path:** superset/charts/schemas.py
**Line:** 1361:1361
**Comment:**
*Custom Rule: Add an explicit type annotation to this newly introduced schema field attribute so the new code complies with the repository's type-hint requirement.
Validate the correctness of the flagged issue. If correct, How can I resolve this? If you propose a fix, implement it and please make it concise.
Once fix is implemented, also check other comments on the same PR, and ask user if the user wants to fix the rest of the comments as well. if said yes, then fetch all the comments validate the correctness and implement a minimal fix
```
</details>
<a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F41184&comment_hash=ca1a3461756e5c59e901da112f4d4a9aaa2f74f2fa09037a7224b84870a95a2e&reaction=like'>👍</a> | <a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F41184&comment_hash=ca1a3461756e5c59e901da112f4d4a9aaa2f74f2fa09037a7224b84870a95a2e&reaction=dislike'>👎</a>
##########
superset/common/query_context_processor.py:
##########
@@ -252,9 +254,54 @@ def get_query_result(self, query_object: QueryObject) -> QueryResult:
This method delegates to the datasource's get_query_result method,
which handles query execution, normalization, time offsets, and
post-processing.
+
+ When the query requests rollup ``grouping_sets`` but the engine does not
+ support native ``GROUPING SETS``, fall back to one query per level and
+ concatenate the results with ``GROUPING()``-equivalent markers, so the
+ combined result matches the shape the native path produces (SIP.md,
+ phase 3b). Engines that support it run the single native query.
"""
+ if query_object.grouping_sets and not self._supports_grouping_sets():
+ return self._grouping_sets_fallback(query_object)
return self._qc_datasource.get_query_result(query_object)
+ def _supports_grouping_sets(self) -> bool:
+ engine_spec = getattr(self._qc_datasource, "db_engine_spec", None)
Review Comment:
**Suggestion:** Add an explicit type annotation for this local variable to comply with the requirement for annotating relevant variables in new code. [custom_rule]
**Severity Level:** Minor 🧹
<details>
<summary><b>Why it matters? ⭐ </b></summary>
This is new Python code introducing a local variable whose type can be annotated, and it lacks an explicit type hint. That matches the custom rule requiring type hints on relevant variables in modified Python code.
</details>
<details>
<summary><b>Rule source 📖 </b></summary>
.cursor/rules/dev-standard.mdc (line 28)
</details>
[](https://app.codeant.ai/fix-in-ide?tool=cursor&prompt_id=997647b0a4134885b8f8ba8ac6a3d7ce&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset) [](https://app.codeant.ai/fix-in-ide?tool=vscode-claude&prompt_id=997647b0a4134885b8f8ba8ac6a3d7ce&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset)
*(Use Cmd/Ctrl + Click for best experience)*
<details>
<summary><b>Prompt for AI Agent 🤖 </b></summary>
```mdx
This is a comment left during a code review.
**Path:** superset/common/query_context_processor.py
**Line:** 269:269
**Comment:**
*Custom Rule: Add an explicit type annotation for this local variable to comply with the requirement for annotating relevant variables in new code.
Validate the correctness of the flagged issue. If correct, How can I resolve this? If you propose a fix, implement it and please make it concise.
Once fix is implemented, also check other comments on the same PR, and ask user if the user wants to fix the rest of the comments as well. if said yes, then fetch all the comments validate the correctness and implement a minimal fix
```
</details>
<a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F41184&comment_hash=561944dd7f2fdf1559f5aaccfdf501c01338bca9453b07972a4ec2660a453ba3&reaction=like'>👍</a> | <a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F41184&comment_hash=561944dd7f2fdf1559f5aaccfdf501c01338bca9453b07972a4ec2660a453ba3&reaction=dislike'>👎</a>
##########
superset/db_engine_specs/snowflake.py:
##########
@@ -96,6 +96,7 @@ class SnowflakeEngineSpec(PostgresBaseEngineSpec):
supports_dynamic_schema = True
supports_catalog = supports_dynamic_catalog = supports_cross_catalog_queries = True
+ supports_grouping_sets = True
Review Comment:
**Suggestion:** Add an explicit type annotation for this new class attribute so the newly introduced variable is type-hinted. [custom_rule]
**Severity Level:** Minor 🧹
<details>
<summary><b>Why it matters? ⭐ </b></summary>
The added class attribute is a boolean constant that can be explicitly annotated, e.g. `supports_grouping_sets: bool = True`. Because the rule requires type hints on new or modified Python code where relevant variables can be annotated, this is a real violation.
</details>
<details>
<summary><b>Rule source 📖 </b></summary>
.cursor/rules/dev-standard.mdc (line 28)
</details>
[](https://app.codeant.ai/fix-in-ide?tool=cursor&prompt_id=40128bd2bc22478c881db795e7673f35&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset) [](https://app.codeant.ai/fix-in-ide?tool=vscode-claude&prompt_id=40128bd2bc22478c881db795e7673f35&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset)
*(Use Cmd/Ctrl + Click for best experience)*
<details>
<summary><b>Prompt for AI Agent 🤖 </b></summary>
```mdx
This is a comment left during a code review.
**Path:** superset/db_engine_specs/snowflake.py
**Line:** 99:99
**Comment:**
*Custom Rule: Add an explicit type annotation for this new class attribute so the newly introduced variable is type-hinted.
Validate the correctness of the flagged issue. If correct, How can I resolve this? If you propose a fix, implement it and please make it concise.
Once fix is implemented, also check other comments on the same PR, and ask user if the user wants to fix the rest of the comments as well. if said yes, then fetch all the comments validate the correctness and implement a minimal fix
```
</details>
<a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F41184&comment_hash=1b022f9c89688ea147a3d349f7f8e9b9fc61e4e2762876b2b35516a5472902dc&reaction=like'>👍</a> | <a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F41184&comment_hash=1b022f9c89688ea147a3d349f7f8e9b9fc61e4e2762876b2b35516a5472902dc&reaction=dislike'>👎</a>
##########
superset/db_engine_specs/postgres.py:
##########
@@ -278,6 +278,7 @@ class PostgresEngineSpec(BasicParametersMixin, PostgresBaseEngineSpec):
supports_dynamic_schema = True
supports_catalog = True
supports_dynamic_catalog = True
+ supports_grouping_sets = True
Review Comment:
**Suggestion:** Add an explicit boolean type annotation to this newly introduced class capability flag. [custom_rule]
**Severity Level:** Minor 🧹
<details>
<summary><b>Why it matters? ⭐ </b></summary>
The new class attribute is a boolean capability flag and is directly annotatable (for example, `supports_grouping_sets: bool = True`). Because the added code omits the type hint, it matches the rule requiring type hints for new or modified Python variables that can be annotated.
</details>
<details>
<summary><b>Rule source 📖 </b></summary>
.cursor/rules/dev-standard.mdc (line 28)
</details>
[](https://app.codeant.ai/fix-in-ide?tool=cursor&prompt_id=07fc29f6a4824a3e8c125b5af41c5fe6&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset) [](https://app.codeant.ai/fix-in-ide?tool=vscode-claude&prompt_id=07fc29f6a4824a3e8c125b5af41c5fe6&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset)
*(Use Cmd/Ctrl + Click for best experience)*
<details>
<summary><b>Prompt for AI Agent 🤖 </b></summary>
```mdx
This is a comment left during a code review.
**Path:** superset/db_engine_specs/postgres.py
**Line:** 281:281
**Comment:**
*Custom Rule: Add an explicit boolean type annotation to this newly introduced class capability flag.
Validate the correctness of the flagged issue. If correct, How can I resolve this? If you propose a fix, implement it and please make it concise.
Once fix is implemented, also check other comments on the same PR, and ask user if the user wants to fix the rest of the comments as well. if said yes, then fetch all the comments validate the correctness and implement a minimal fix
```
</details>
<a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F41184&comment_hash=0c734684148f6c863613b08aa02fb20ba713b55b978ee126fd7b7f5391546c4b&reaction=like'>👍</a> | <a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F41184&comment_hash=0c734684148f6c863613b08aa02fb20ba713b55b978ee126fd7b7f5391546c4b&reaction=dislike'>👎</a>
##########
superset-frontend/plugins/plugin-chart-pivot-table/test/plugin/utilities.test.ts:
##########
@@ -0,0 +1,446 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import { TimeGranularity } from '@superset-ui/core';
+import buildGroupbyCombinations, {
+ isAdditiveMetric,
+ allMetricsAdditive,
+ additiveReducerFor,
+ synthesizeAdditiveLevels,
+ splitGroupingSetsResult,
+ groupingMarkerLabel,
+} from '../../src/plugin/utilities';
+import { PivotTableQueryFormData, MetricsLayoutEnum } from '../../src/types';
+
+const baseFormData = {
+ groupbyRows: ['row1', 'row2'],
+ groupbyColumns: ['col1', 'col2'],
+ metrics: ['metric1', 'metric2'],
+ tableRenderer: 'Table With Subtotal',
+ colOrder: 'key_a_to_z',
+ rowOrder: 'key_a_to_z',
+ aggregateFunction: 'Sum',
+ metricsLayout: MetricsLayoutEnum.ROWS,
+ transposePivot: false,
+ rowSubtotalPosition: true,
+ colSubtotalPosition: true,
+ colTotals: true,
+ colSubTotals: true,
+ rowTotals: true,
+ rowSubTotals: true,
+ valueFormat: 'SMART_NUMBER',
+ datasource: '5__table',
+ viz_type: 'my_chart',
+ width: 800,
+ height: 600,
+ combineMetric: false,
+ verboseMap: {},
+ columnFormats: {},
+ currencyFormats: {},
+ metricColorFormatters: [],
+ dateFormatters: {},
+ setDataMask: () => {},
+ legacy_order_by: 'count',
+ order_desc: true,
+ margin: 0,
+ time_grain_sqla: TimeGranularity.MONTH,
+ temporal_columns_lookup: { col1: true },
+ currencyFormat: { symbol: 'USD', symbolPosition: 'prefix' },
+} as unknown as PivotTableQueryFormData;
+
+test('should build all combinations for basic pivot table', () => {
+ const combinations = buildGroupbyCombinations(baseFormData);
+
+ expect(combinations).toEqual([
+ { rows: [], columns: [] },
+ { rows: [], columns: ['col1'] },
+ { rows: [], columns: ['col1', 'col2'] },
+ { rows: ['row1'], columns: [] },
+ { rows: ['row1'], columns: ['col1'] },
+ { rows: ['row1'], columns: ['col1', 'col2'] },
+ { rows: ['row1', 'row2'], columns: [] },
+ { rows: ['row1', 'row2'], columns: ['col1'] },
+ { rows: ['row1', 'row2'], columns: ['col1', 'col2'] },
+ ]);
+
+ expect(combinations).toHaveLength(9);
+});
+
+test('should handle transposed pivot correctly', () => {
+ const modifiedFormData = {
+ ...baseFormData,
+ transposePivot: true,
+ };
+
+ const combinations = buildGroupbyCombinations(modifiedFormData);
+
+ expect(combinations).toEqual([
+ { rows: [], columns: [] },
+ { rows: [], columns: ['row1'] },
+ { rows: [], columns: ['row1', 'row2'] },
+ { rows: ['col1'], columns: [] },
+ { rows: ['col1'], columns: ['row1'] },
+ { rows: ['col1'], columns: ['row1', 'row2'] },
+ { rows: ['col1', 'col2'], columns: [] },
+ { rows: ['col1', 'col2'], columns: ['row1'] },
+ { rows: ['col1', 'col2'], columns: ['row1', 'row2'] },
+ ]);
+});
+
+test('should filter combinations when combineMetric is true with ROWS layout', () => {
+ const modifiedFormData = {
+ ...baseFormData,
+ combineMetric: true,
+ metricsLayout: MetricsLayoutEnum.ROWS,
+ };
+
+ const combinations = buildGroupbyCombinations(modifiedFormData);
+
+ expect(combinations).toEqual([
+ { rows: ['row1', 'row2'], columns: [] },
+ { rows: ['row1', 'row2'], columns: ['col1'] },
+ { rows: ['row1', 'row2'], columns: ['col1', 'col2'] },
+ ]);
+
+ expect(combinations).toHaveLength(3);
+});
+
+test('should filter combinations when combineMetric is true with COLUMNS layout', () => {
+ const modifiedFormData = {
+ ...baseFormData,
+ combineMetric: true,
+ metricsLayout: MetricsLayoutEnum.COLUMNS,
+ };
+
+ const combinations = buildGroupbyCombinations(modifiedFormData);
+
+ expect(combinations).toEqual([
+ { rows: [], columns: ['col1', 'col2'] },
+ { rows: ['row1'], columns: ['col1', 'col2'] },
+ { rows: ['row1', 'row2'], columns: ['col1', 'col2'] },
+ ]);
+
+ expect(combinations).toHaveLength(3);
+});
+
+test('should handle single dimension in rows only', () => {
+ const modifiedFormData = {
+ ...baseFormData,
+ groupbyRows: ['row'],
+ groupbyColumns: [],
+ };
+
+ const combinations = buildGroupbyCombinations(modifiedFormData);
+
+ expect(combinations).toEqual([
+ { rows: [], columns: [] },
+ { rows: ['row'], columns: [] },
+ ]);
+
+ expect(combinations).toHaveLength(2);
+});
+
+test('should handle single dimension in columns only', () => {
+ const modifiedFormData = {
+ ...baseFormData,
+ groupbyRows: [],
+ groupbyColumns: ['col'],
+ };
+
+ const combinations = buildGroupbyCombinations(modifiedFormData);
+
+ expect(combinations).toEqual([
+ { rows: [], columns: [] },
+ { rows: [], columns: ['col'] },
+ ]);
+
+ expect(combinations).toHaveLength(2);
+});
+
+test('should handle empty groupby arrays', () => {
+ const modifiedFormData = {
+ ...baseFormData,
+ groupbyRows: [],
+ groupbyColumns: [],
+ };
+
+ const combinations = buildGroupbyCombinations(modifiedFormData);
+
+ expect(combinations).toEqual([{ rows: [], columns: [] }]);
+
+ expect(combinations).toHaveLength(1);
+});
+
+test('should work with combineMetric and transposed pivot', () => {
+ const modifiedFormData = {
+ ...baseFormData,
+ transposePivot: true,
+ combineMetric: true,
+ metricsLayout: MetricsLayoutEnum.COLUMNS,
+ };
+
+ const combinations = buildGroupbyCombinations(modifiedFormData);
+
+ expect(combinations).toEqual([
+ { rows: [], columns: ['row1', 'row2'] },
+ { rows: ['col1'], columns: ['row1', 'row2'] },
+ { rows: ['col1', 'col2'], columns: ['row1', 'row2'] },
+ ]);
+});
+
+test('should handle combineMetric with empty arrays correctly', () => {
+ const modifiedFormData = {
+ ...baseFormData,
+ groupbyRows: [],
+ groupbyColumns: ['col'],
+ combineMetric: true,
+ metricsLayout: MetricsLayoutEnum.ROWS,
+ };
+
+ const combinations = buildGroupbyCombinations(modifiedFormData);
+
+ expect(combinations).toEqual([
+ { rows: [], columns: [] },
+ { rows: [], columns: ['col'] },
+ ]);
+});
+
+test('should work with large number of dimensions', () => {
+ const modifiedFormData = {
+ ...baseFormData,
+ groupbyRows: ['r1', 'r2', 'r3', 'r4'],
+ groupbyColumns: ['c1', 'c2', 'c3'],
+ };
+
+ const combinations = buildGroupbyCombinations(modifiedFormData);
+
+ expect(combinations).toHaveLength(20);
+
+ expect(combinations).toContainEqual({ rows: [], columns: [] });
+ expect(combinations).toContainEqual({
+ rows: ['r1', 'r2', 'r3', 'r4'],
+ columns: ['c1', 'c2', 'c3'],
+ });
+});
+
+test('isAdditiveMetric: SIMPLE metrics with additive aggregates are additive', () => {
+ expect(
+ isAdditiveMetric({
+ expressionType: 'SIMPLE',
+ aggregate: 'SUM',
+ column: { column_name: 'num' },
+ label: 'sum_num',
+ } as any),
+ ).toBe(true);
+ expect(
+ isAdditiveMetric({
+ expressionType: 'SIMPLE',
+ aggregate: 'COUNT',
+ column: { column_name: 'num' },
+ label: 'count_num',
+ } as any),
Review Comment:
**Suggestion:** Replace this `any` cast with the proper metric object type instead of forcing the value through `any`. [custom_rule]
**Severity Level:** Minor 🧹
<details>
<summary><b>Why it matters? ⭐ </b></summary>
This line also uses an `any` cast in new TypeScript test code, so it violates the no-any-types rule.
</details>
<details>
<summary><b>Rule source 📖 </b></summary>
.cursor/rules/dev-standard.mdc (line 16)
</details>
[](https://app.codeant.ai/fix-in-ide?tool=cursor&prompt_id=1333e298f0a3414ab46c2be0f55d521c&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset) [](https://app.codeant.ai/fix-in-ide?tool=vscode-claude&prompt_id=1333e298f0a3414ab46c2be0f55d521c&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset)
*(Use Cmd/Ctrl + Click for best experience)*
<details>
<summary><b>Prompt for AI Agent 🤖 </b></summary>
```mdx
This is a comment left during a code review.
**Path:** superset-frontend/plugins/plugin-chart-pivot-table/test/plugin/utilities.test.ts
**Line:** 257:257
**Comment:**
*Custom Rule: Replace this `any` cast with the proper metric object type instead of forcing the value through `any`.
Validate the correctness of the flagged issue. If correct, How can I resolve this? If you propose a fix, implement it and please make it concise.
Once fix is implemented, also check other comments on the same PR, and ask user if the user wants to fix the rest of the comments as well. if said yes, then fetch all the comments validate the correctness and implement a minimal fix
```
</details>
<a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F41184&comment_hash=a1d9a7f9c78b90a1c411074a4dcfa05dede63b037f545967fd8065061ac644e6&reaction=like'>👍</a> | <a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F41184&comment_hash=a1d9a7f9c78b90a1c411074a4dcfa05dede63b037f545967fd8065061ac644e6&reaction=dislike'>👎</a>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(table/pivot-table): correct non-additive totals/subtotals via DB rollup [POC for SIP-216] [superset]
Posted by "codeant-ai-for-open-source[bot] (via GitHub)" <gi...@apache.org>.
codeant-ai-for-open-source[bot] commented on code in PR #41184:
URL: https://github.com/apache/superset/pull/41184#discussion_r3555846902
##########
superset-frontend/plugins/plugin-chart-pivot-table/src/PivotTableChart.tsx:
##########
@@ -341,22 +296,43 @@ export default function PivotTableChart(props: PivotTableProps) {
const unpivotedData = useMemo(
() =>
- data.reduce(
- (acc: Record<string, any>[], record: Record<string, any>) => [
- ...acc,
- ...metricNames
+ // `data` is now one entry per rollup level. Tag every record with the
+ // row/column dimension labels of the level that produced it (mirroring
+ // the METRIC_KEY injection used for the full rows/cols below) so
+ // PivotData can slot each pre-computed value without re-aggregating.
+ // buildGroupbyCombinations already applied transposePivot, so the level's
+ // groupby is display-oriented and is not transposed again here.
+ data.flatMap((query: QueryData) => {
+ let levelRows = query.groupby.rows.map(getColumnLabel);
+ let levelCols = query.groupby.columns.map(getColumnLabel);
+ if (metricsLayout === MetricsLayoutEnum.ROWS) {
+ levelRows = combineMetric
+ ? [...levelRows, METRIC_KEY]
+ : [METRIC_KEY, ...levelRows];
+ } else {
+ levelCols = combineMetric
+ ? [...levelCols, METRIC_KEY]
+ : [METRIC_KEY, ...levelCols];
+ }
+ return query.data.flatMap((record: Record<string, any>) =>
Review Comment:
**Suggestion:** Replace the `any` value type in this record annotation with a concrete union or existing data-row type so the transformed pivot records remain fully type-safe. [custom_rule]
**Severity Level:** Minor ⚠️
<details>
<summary><b>Why it matters? 🤔 </b></summary>
The final file still contains `Record<string, any>` in a changed TypeScript hunk, which directly violates the no-any-types rule. The suggestion accurately identifies this real issue.
</details>
<details>
<summary><b>Rule source 📖 </b></summary>
.cursor/rules/dev-standard.mdc (line 16)
</details>
[](https://app.codeant.ai/fix-in-ide?tool=cursor&prompt_id=c42d6951145f4659a53d1cd54369d21e&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset) [](https://app.codeant.ai/fix-in-ide?tool=vscode-claude&prompt_id=c42d6951145f4659a53d1cd54369d21e&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset)
*(Use Cmd/Ctrl + Click for best experience)*
<details>
<summary><b>Prompt for AI Agent 🤖 </b></summary>
```mdx
This is a comment left during a code review.
**Path:** superset-frontend/plugins/plugin-chart-pivot-table/src/PivotTableChart.tsx
**Line:** 317:317
**Comment:**
*Custom Rule: Replace the `any` value type in this record annotation with a concrete union or existing data-row type so the transformed pivot records remain fully type-safe.
Validate the correctness of the flagged issue. If correct, How can I resolve this? If you propose a fix, implement it and please make it concise.
Once fix is implemented, also check other comments on the same PR, and ask user if the user wants to fix the rest of the comments as well. if said yes, then fetch all the comments validate the correctness and implement a minimal fix
```
</details>
<a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F41184&comment_hash=e8199e3de377a1926960159749a408e45366de0f7b04125f7116ab0bddae1fa9&reaction=like'>👍</a> | <a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F41184&comment_hash=e8199e3de377a1926960159749a408e45366de0f7b04125f7116ab0bddae1fa9&reaction=dislike'>👎</a>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(table/pivot-table): correct non-additive totals/subtotals via DB rollup [POC for SIP-216] [superset]
Posted by "rusackas (via GitHub)" <gi...@apache.org>.
rusackas commented on code in PR #41184:
URL: https://github.com/apache/superset/pull/41184#discussion_r3557744328
##########
superset/common/query_object.py:
##########
@@ -157,6 +158,7 @@ def __init__( # pylint: disable=too-many-locals, too-many-arguments
self.series_limit = series_limit
self.series_limit_metric = series_limit_metric
self.group_others_when_limit_reached = group_others_when_limit_reached
+ self.grouping_sets = grouping_sets or []
Review Comment:
Good call, added `grouping_sets: list[list[str]]` to the class attrs alongside the others.
##########
superset-frontend/plugins/plugin-chart-pivot-table/test/plugin/buildQuery.test.ts:
##########
@@ -56,9 +55,41 @@ const formData: PivotTableQueryFormData = {
currencyFormat: { symbol: 'USD', symbolPosition: 'prefix' },
};
-test('should build groupby with series in form data', () => {
+test('additive metrics use the fast-path: a single full-detail query', () => {
+ const { queries } = buildQuery({
+ ...formData,
+ metrics: [
+ {
+ expressionType: 'SIMPLE',
+ aggregate: 'SUM',
+ column: { column_name: 'num' },
+ label: 'sum_num',
+ },
+ ] as any,
+ });
+ expect(queries).toHaveLength(1);
+ // The single leaf query carries all dimensions (2 rows + 2 cols).
+ expect(queries[0].columns).toHaveLength(4);
+});
+
+test('non-additive metrics emit a single GROUPING SETS query with all levels', () => {
+ // 2 row dims x 2 col dims -> (2+1) x (2+1) = 9 rollup levels, carried as
+ // grouping_sets on a single query (saved-metric strings are non-additive).
const queryContext = buildQuery(formData);
+ expect(queryContext.queries).toHaveLength(1);
const [query] = queryContext.queries;
+ // The single query selects the full set of dimensions ...
+ expect(query.columns).toHaveLength(4);
+ // ... and requests every rollup level via grouping_sets (grand total = []).
+ expect((query as any).grouping_sets).toHaveLength(9);
+ expect((query as any).grouping_sets[0]).toEqual([]);
+ expect((query as any).grouping_sets[8]).toHaveLength(4);
+});
+
+test('should build groupby with series in form data', () => {
+ const queryContext = buildQuery(formData);
+ // Multi-query rollup: the full-detail level is the last query (grand total is first).
+ const query = queryContext.queries[queryContext.queries.length - 1];
Review Comment:
Yep, stale wording from an earlier multi-query iteration. Updated the comment to reflect the single GROUPING SETS query.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(table/pivot-table): correct non-additive totals/subtotals via DB rollup [POC for SIP-216] [superset]
Posted by "rusackas (via GitHub)" <gi...@apache.org>.
rusackas commented on code in PR #41184:
URL: https://github.com/apache/superset/pull/41184#discussion_r3565611545
##########
superset-frontend/plugins/plugin-chart-pivot-table/src/plugin/buildQuery.ts:
##########
@@ -54,6 +59,39 @@ export default function buildQuery(formData: PivotTableQueryFormData) {
}
return col;
});
+}
+
+export default function buildQuery(formData: PivotTableQueryFormData) {
+ const { extra_form_data } = formData;
+ const time_grain_sqla =
+ extra_form_data?.time_grain_sqla || formData.time_grain_sqla;
+
+ // The full set of dimensions (the leaf level) is always queried; the rollup
+ // levels are derived from it. Apply transposePivot here exactly as
+ // buildGroupbyCombinations does, so the column order matches the rollup
+ // levels. See SIP.md.
+ const [fullRows, fullColumnDims] = formData.transposePivot
+ ? [formData.groupbyColumns, formData.groupbyRows]
+ : [formData.groupbyRows, formData.groupbyColumns];
+ const fullGroupby: Groupby = {
+ rows: ensureIsArray<QueryFormColumn>(fullRows),
+ columns: ensureIsArray<QueryFormColumn>(fullColumnDims),
+ };
+ const fullColumns = getQueryColumns(fullGroupby, formData, time_grain_sqla);
+
+ // A single query always suffices:
+ // - additive metrics (SUM/COUNT/MIN/MAX): a full-detail query whose
+ // subtotals/grand totals transformProps derives by client-side reduction.
+ // - non-additive metrics: a GROUPING SETS query (one `grouping_sets` level
+ // per displayed total/subtotal) so the database computes every level; the
+ // backend falls back to per-level queries on engines without native
+ // support. transformProps splits the combined result by level.
+ const additive = allMetricsAdditive(ensureIsArray(formData.metrics));
+ const groupingSets = additive
+ ? undefined
+ : buildGroupbyCombinations(formData).map(level =>
+ [...level.rows, ...level.columns].map(getColumnLabel),
Review Comment:
Good catch, deduped the labels per level. A dimension on both axes would otherwise land twice in the same GROUPING SETS tuple.
##########
superset-frontend/plugins/plugin-chart-pivot-table/test/plugin/utilities.test.ts:
##########
@@ -0,0 +1,446 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import { TimeGranularity } from '@superset-ui/core';
+import buildGroupbyCombinations, {
+ isAdditiveMetric,
+ allMetricsAdditive,
+ additiveReducerFor,
+ synthesizeAdditiveLevels,
+ splitGroupingSetsResult,
+ groupingMarkerLabel,
+} from '../../src/plugin/utilities';
+import { PivotTableQueryFormData, MetricsLayoutEnum } from '../../src/types';
+
+const baseFormData = {
+ groupbyRows: ['row1', 'row2'],
+ groupbyColumns: ['col1', 'col2'],
+ metrics: ['metric1', 'metric2'],
+ tableRenderer: 'Table With Subtotal',
+ colOrder: 'key_a_to_z',
+ rowOrder: 'key_a_to_z',
+ aggregateFunction: 'Sum',
+ metricsLayout: MetricsLayoutEnum.ROWS,
+ transposePivot: false,
+ rowSubtotalPosition: true,
+ colSubtotalPosition: true,
+ colTotals: true,
+ colSubTotals: true,
+ rowTotals: true,
+ rowSubTotals: true,
+ valueFormat: 'SMART_NUMBER',
+ datasource: '5__table',
+ viz_type: 'my_chart',
+ width: 800,
+ height: 600,
+ combineMetric: false,
+ verboseMap: {},
+ columnFormats: {},
+ currencyFormats: {},
+ metricColorFormatters: [],
+ dateFormatters: {},
+ setDataMask: () => {},
+ legacy_order_by: 'count',
+ order_desc: true,
+ margin: 0,
+ time_grain_sqla: TimeGranularity.MONTH,
+ temporal_columns_lookup: { col1: true },
+ currencyFormat: { symbol: 'USD', symbolPosition: 'prefix' },
+} as unknown as PivotTableQueryFormData;
+
+test('should build all combinations for basic pivot table', () => {
+ const combinations = buildGroupbyCombinations(baseFormData);
+
+ expect(combinations).toEqual([
+ { rows: [], columns: [] },
+ { rows: [], columns: ['col1'] },
+ { rows: [], columns: ['col1', 'col2'] },
+ { rows: ['row1'], columns: [] },
+ { rows: ['row1'], columns: ['col1'] },
+ { rows: ['row1'], columns: ['col1', 'col2'] },
+ { rows: ['row1', 'row2'], columns: [] },
+ { rows: ['row1', 'row2'], columns: ['col1'] },
+ { rows: ['row1', 'row2'], columns: ['col1', 'col2'] },
+ ]);
+
+ expect(combinations).toHaveLength(9);
+});
+
+test('should handle transposed pivot correctly', () => {
+ const modifiedFormData = {
+ ...baseFormData,
+ transposePivot: true,
+ };
+
+ const combinations = buildGroupbyCombinations(modifiedFormData);
+
+ expect(combinations).toEqual([
+ { rows: [], columns: [] },
+ { rows: [], columns: ['row1'] },
+ { rows: [], columns: ['row1', 'row2'] },
+ { rows: ['col1'], columns: [] },
+ { rows: ['col1'], columns: ['row1'] },
+ { rows: ['col1'], columns: ['row1', 'row2'] },
+ { rows: ['col1', 'col2'], columns: [] },
+ { rows: ['col1', 'col2'], columns: ['row1'] },
+ { rows: ['col1', 'col2'], columns: ['row1', 'row2'] },
+ ]);
+});
+
+test('should filter combinations when combineMetric is true with ROWS layout', () => {
+ const modifiedFormData = {
+ ...baseFormData,
+ combineMetric: true,
+ metricsLayout: MetricsLayoutEnum.ROWS,
+ };
+
+ const combinations = buildGroupbyCombinations(modifiedFormData);
+
+ expect(combinations).toEqual([
+ { rows: ['row1', 'row2'], columns: [] },
+ { rows: ['row1', 'row2'], columns: ['col1'] },
+ { rows: ['row1', 'row2'], columns: ['col1', 'col2'] },
+ ]);
+
+ expect(combinations).toHaveLength(3);
+});
+
+test('should filter combinations when combineMetric is true with COLUMNS layout', () => {
+ const modifiedFormData = {
+ ...baseFormData,
+ combineMetric: true,
+ metricsLayout: MetricsLayoutEnum.COLUMNS,
+ };
+
+ const combinations = buildGroupbyCombinations(modifiedFormData);
+
+ expect(combinations).toEqual([
+ { rows: [], columns: ['col1', 'col2'] },
+ { rows: ['row1'], columns: ['col1', 'col2'] },
+ { rows: ['row1', 'row2'], columns: ['col1', 'col2'] },
+ ]);
+
+ expect(combinations).toHaveLength(3);
+});
+
+test('should handle single dimension in rows only', () => {
+ const modifiedFormData = {
+ ...baseFormData,
+ groupbyRows: ['row'],
+ groupbyColumns: [],
+ };
+
+ const combinations = buildGroupbyCombinations(modifiedFormData);
+
+ expect(combinations).toEqual([
+ { rows: [], columns: [] },
+ { rows: ['row'], columns: [] },
+ ]);
+
+ expect(combinations).toHaveLength(2);
+});
+
+test('should handle single dimension in columns only', () => {
+ const modifiedFormData = {
+ ...baseFormData,
+ groupbyRows: [],
+ groupbyColumns: ['col'],
+ };
+
+ const combinations = buildGroupbyCombinations(modifiedFormData);
+
+ expect(combinations).toEqual([
+ { rows: [], columns: [] },
+ { rows: [], columns: ['col'] },
+ ]);
+
+ expect(combinations).toHaveLength(2);
+});
+
+test('should handle empty groupby arrays', () => {
+ const modifiedFormData = {
+ ...baseFormData,
+ groupbyRows: [],
+ groupbyColumns: [],
+ };
+
+ const combinations = buildGroupbyCombinations(modifiedFormData);
+
+ expect(combinations).toEqual([{ rows: [], columns: [] }]);
+
+ expect(combinations).toHaveLength(1);
+});
+
+test('should work with combineMetric and transposed pivot', () => {
+ const modifiedFormData = {
+ ...baseFormData,
+ transposePivot: true,
+ combineMetric: true,
+ metricsLayout: MetricsLayoutEnum.COLUMNS,
+ };
+
+ const combinations = buildGroupbyCombinations(modifiedFormData);
+
+ expect(combinations).toEqual([
+ { rows: [], columns: ['row1', 'row2'] },
+ { rows: ['col1'], columns: ['row1', 'row2'] },
+ { rows: ['col1', 'col2'], columns: ['row1', 'row2'] },
+ ]);
+});
+
+test('should handle combineMetric with empty arrays correctly', () => {
+ const modifiedFormData = {
+ ...baseFormData,
+ groupbyRows: [],
+ groupbyColumns: ['col'],
+ combineMetric: true,
+ metricsLayout: MetricsLayoutEnum.ROWS,
+ };
+
+ const combinations = buildGroupbyCombinations(modifiedFormData);
+
+ expect(combinations).toEqual([
+ { rows: [], columns: [] },
+ { rows: [], columns: ['col'] },
+ ]);
+});
+
+test('should work with large number of dimensions', () => {
+ const modifiedFormData = {
+ ...baseFormData,
+ groupbyRows: ['r1', 'r2', 'r3', 'r4'],
+ groupbyColumns: ['c1', 'c2', 'c3'],
+ };
+
+ const combinations = buildGroupbyCombinations(modifiedFormData);
+
+ expect(combinations).toHaveLength(20);
+
+ expect(combinations).toContainEqual({ rows: [], columns: [] });
+ expect(combinations).toContainEqual({
+ rows: ['r1', 'r2', 'r3', 'r4'],
+ columns: ['c1', 'c2', 'c3'],
+ });
+});
+
+test('isAdditiveMetric: SIMPLE metrics with additive aggregates are additive', () => {
+ expect(
+ isAdditiveMetric({
+ expressionType: 'SIMPLE',
+ aggregate: 'SUM',
+ column: { column_name: 'num' },
+ label: 'sum_num',
+ } as any),
+ ).toBe(true);
+ expect(
+ isAdditiveMetric({
+ expressionType: 'SIMPLE',
+ aggregate: 'COUNT',
+ column: { column_name: 'num' },
+ label: 'count_num',
+ } as any),
+ ).toBe(true);
+});
+
+test('isAdditiveMetric: non-additive aggregates, SQL, and saved metrics are not additive', () => {
+ expect(
+ isAdditiveMetric({
+ expressionType: 'SIMPLE',
+ aggregate: 'AVG',
+ column: { column_name: 'num' },
+ label: 'avg_num',
+ } as any),
+ ).toBe(false);
+ expect(
+ isAdditiveMetric({
+ expressionType: 'SIMPLE',
+ aggregate: 'COUNT_DISTINCT',
+ column: { column_name: 'name' },
+ label: 'distinct_names',
+ } as any),
+ ).toBe(false);
+ expect(
+ isAdditiveMetric({
+ expressionType: 'SQL',
+ sqlExpression: 'SUM(a) / SUM(b)',
+ label: 'ratio',
+ } as any),
+ ).toBe(false);
+ // saved-metric reference: aggregate unknown from form data -> non-additive
+ expect(isAdditiveMetric('count' as any)).toBe(false);
Review Comment:
Fixed, `isAdditiveMetric` already accepts a string so the cast was unneeded.
##########
superset/common/grouping_sets.py:
##########
@@ -0,0 +1,117 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""
+SQL building blocks for the pivot-table non-additive totals optimization
+(SIP.md, phase 3b). When a datasource engine reports
+``supports_grouping_sets``, the N per-rollup-level queries can be collapsed into
+a single ``GROUPING SETS`` query: the database computes every level in one scan,
+and each returned row is attributed to its level via ``GROUPING()`` markers.
+
+These are the engine-agnostic SQL primitives. Wiring them into the query context
+(emitting one query and splitting the result back into per-level results) is the
+remaining integration; see SIP.md.
+"""
+
+from __future__ import annotations
+
+from collections.abc import Sequence
+from typing import Final
+
+import pandas as pd
+from sqlalchemy import func, tuple_
+from sqlalchemy.sql.elements import ColumnElement
+
+# Suffix for the per-column GROUPING() marker columns added to a GROUPING SETS
+# query. Chosen to be unlikely to collide with a real metric/column label.
+GROUPING_MARKER_SUFFIX: Final = "__superset_grouping"
+
+
+def grouping_marker_label(column_label: str) -> str:
+ """The output label of the GROUPING() marker for a groupby column."""
+ return f"{column_label}{GROUPING_MARKER_SUFFIX}"
+
+
+def grouping_sets_clause(
+ groups: Sequence[Sequence[ColumnElement]],
+) -> ColumnElement:
+ """
+ Build a ``GROUP BY GROUPING SETS (...)`` clause from rollup column groups.
+
+ Each group is the set of columns grouped at one rollup level; the empty
+ group ``()`` is the grand total. For example, groups ``[[a, b], [a], []]``
+ produce ``GROUPING SETS ((a, b), (a), ())``.
+
+ :param groups: one column list per rollup level
+ :return: a clause element suitable for ``select(...).group_by(...)``
+ """
+ return func.grouping_sets(*[tuple_(*group) for group in groups])
+
+
+def grouping_id_column(column: ColumnElement, label: str) -> ColumnElement:
+ """
+ Build a ``GROUPING(col) AS label`` marker column.
+
+ In a ``GROUPING SETS`` result, ``GROUPING(col)`` is ``0`` when ``col`` is
+ part of the row's grouping level and ``1`` when it has been rolled up
+ (aggregated away). Selecting one marker per groupby column lets the caller
+ attribute each returned row to its rollup level when splitting the single
+ result back into per-level results.
+
+ :param column: the groupby column to probe
+ :param label: the output label for the marker (see ``grouping_marker_label``)
+ :return: the labelled ``GROUPING(col)`` column
+ """
+ return func.grouping(column).label(label)
+
+
+def split_grouping_sets_result(
+ df: pd.DataFrame,
+ levels: Sequence[Sequence[str]],
+ groupby_columns: Sequence[str],
+) -> list[pd.DataFrame]:
+ """
+ Split a combined ``GROUPING SETS`` result into one DataFrame per rollup
+ level, the inverse of {@link grouping_sets_clause}.
+
+ Each row of ``df`` carries a ``GROUPING()`` marker per groupby column (named
+ by ``grouping_marker_label``): ``0`` if the column is grouped at that row's
+ level, ``1`` if it was rolled up. A row belongs to a level iff its markers
+ are ``0`` exactly for that level's columns. Marker columns are dropped from
+ the returned frames so each looks like an ordinary per-level query result.
+
+ :param df: the combined query result, including marker columns
+ :param levels: the grouped-column list for each rollup level (same order as
+ passed to ``grouping_sets_clause``)
+ :param groupby_columns: every groupby column that has a marker
+ :return: one DataFrame per level, in ``levels`` order
+ """
+ markers = [grouping_marker_label(col) for col in groupby_columns]
Review Comment:
Added the type hint.
##########
superset/common/grouping_sets.py:
##########
@@ -0,0 +1,117 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""
+SQL building blocks for the pivot-table non-additive totals optimization
+(SIP.md, phase 3b). When a datasource engine reports
+``supports_grouping_sets``, the N per-rollup-level queries can be collapsed into
+a single ``GROUPING SETS`` query: the database computes every level in one scan,
+and each returned row is attributed to its level via ``GROUPING()`` markers.
+
+These are the engine-agnostic SQL primitives. Wiring them into the query context
+(emitting one query and splitting the result back into per-level results) is the
+remaining integration; see SIP.md.
+"""
+
+from __future__ import annotations
+
+from collections.abc import Sequence
+from typing import Final
+
+import pandas as pd
+from sqlalchemy import func, tuple_
+from sqlalchemy.sql.elements import ColumnElement
+
+# Suffix for the per-column GROUPING() marker columns added to a GROUPING SETS
+# query. Chosen to be unlikely to collide with a real metric/column label.
+GROUPING_MARKER_SUFFIX: Final = "__superset_grouping"
+
+
+def grouping_marker_label(column_label: str) -> str:
+ """The output label of the GROUPING() marker for a groupby column."""
+ return f"{column_label}{GROUPING_MARKER_SUFFIX}"
+
+
+def grouping_sets_clause(
+ groups: Sequence[Sequence[ColumnElement]],
+) -> ColumnElement:
+ """
+ Build a ``GROUP BY GROUPING SETS (...)`` clause from rollup column groups.
+
+ Each group is the set of columns grouped at one rollup level; the empty
+ group ``()`` is the grand total. For example, groups ``[[a, b], [a], []]``
+ produce ``GROUPING SETS ((a, b), (a), ())``.
+
+ :param groups: one column list per rollup level
+ :return: a clause element suitable for ``select(...).group_by(...)``
+ """
+ return func.grouping_sets(*[tuple_(*group) for group in groups])
+
+
+def grouping_id_column(column: ColumnElement, label: str) -> ColumnElement:
+ """
+ Build a ``GROUPING(col) AS label`` marker column.
+
+ In a ``GROUPING SETS`` result, ``GROUPING(col)`` is ``0`` when ``col`` is
+ part of the row's grouping level and ``1`` when it has been rolled up
+ (aggregated away). Selecting one marker per groupby column lets the caller
+ attribute each returned row to its rollup level when splitting the single
+ result back into per-level results.
+
+ :param column: the groupby column to probe
+ :param label: the output label for the marker (see ``grouping_marker_label``)
+ :return: the labelled ``GROUPING(col)`` column
+ """
+ return func.grouping(column).label(label)
+
+
+def split_grouping_sets_result(
+ df: pd.DataFrame,
+ levels: Sequence[Sequence[str]],
+ groupby_columns: Sequence[str],
+) -> list[pd.DataFrame]:
+ """
+ Split a combined ``GROUPING SETS`` result into one DataFrame per rollup
+ level, the inverse of {@link grouping_sets_clause}.
+
+ Each row of ``df`` carries a ``GROUPING()`` marker per groupby column (named
+ by ``grouping_marker_label``): ``0`` if the column is grouped at that row's
+ level, ``1`` if it was rolled up. A row belongs to a level iff its markers
+ are ``0`` exactly for that level's columns. Marker columns are dropped from
+ the returned frames so each looks like an ordinary per-level query result.
+
+ :param df: the combined query result, including marker columns
+ :param levels: the grouped-column list for each rollup level (same order as
+ passed to ``grouping_sets_clause``)
+ :param groupby_columns: every groupby column that has a marker
+ :return: one DataFrame per level, in ``levels`` order
+ """
+ markers = [grouping_marker_label(col) for col in groupby_columns]
+ results: list[pd.DataFrame] = []
+ for level in levels:
+ grouped = set(level)
Review Comment:
Added the type hint.
##########
superset/common/grouping_sets.py:
##########
@@ -0,0 +1,117 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""
+SQL building blocks for the pivot-table non-additive totals optimization
+(SIP.md, phase 3b). When a datasource engine reports
+``supports_grouping_sets``, the N per-rollup-level queries can be collapsed into
+a single ``GROUPING SETS`` query: the database computes every level in one scan,
+and each returned row is attributed to its level via ``GROUPING()`` markers.
+
+These are the engine-agnostic SQL primitives. Wiring them into the query context
+(emitting one query and splitting the result back into per-level results) is the
+remaining integration; see SIP.md.
+"""
+
+from __future__ import annotations
+
+from collections.abc import Sequence
+from typing import Final
+
+import pandas as pd
+from sqlalchemy import func, tuple_
+from sqlalchemy.sql.elements import ColumnElement
+
+# Suffix for the per-column GROUPING() marker columns added to a GROUPING SETS
+# query. Chosen to be unlikely to collide with a real metric/column label.
+GROUPING_MARKER_SUFFIX: Final = "__superset_grouping"
+
+
+def grouping_marker_label(column_label: str) -> str:
+ """The output label of the GROUPING() marker for a groupby column."""
+ return f"{column_label}{GROUPING_MARKER_SUFFIX}"
+
+
+def grouping_sets_clause(
+ groups: Sequence[Sequence[ColumnElement]],
+) -> ColumnElement:
+ """
+ Build a ``GROUP BY GROUPING SETS (...)`` clause from rollup column groups.
+
+ Each group is the set of columns grouped at one rollup level; the empty
+ group ``()`` is the grand total. For example, groups ``[[a, b], [a], []]``
+ produce ``GROUPING SETS ((a, b), (a), ())``.
+
+ :param groups: one column list per rollup level
+ :return: a clause element suitable for ``select(...).group_by(...)``
+ """
+ return func.grouping_sets(*[tuple_(*group) for group in groups])
+
+
+def grouping_id_column(column: ColumnElement, label: str) -> ColumnElement:
+ """
+ Build a ``GROUPING(col) AS label`` marker column.
+
+ In a ``GROUPING SETS`` result, ``GROUPING(col)`` is ``0`` when ``col`` is
+ part of the row's grouping level and ``1`` when it has been rolled up
+ (aggregated away). Selecting one marker per groupby column lets the caller
+ attribute each returned row to its rollup level when splitting the single
+ result back into per-level results.
+
+ :param column: the groupby column to probe
+ :param label: the output label for the marker (see ``grouping_marker_label``)
+ :return: the labelled ``GROUPING(col)`` column
+ """
+ return func.grouping(column).label(label)
+
+
+def split_grouping_sets_result(
+ df: pd.DataFrame,
+ levels: Sequence[Sequence[str]],
+ groupby_columns: Sequence[str],
+) -> list[pd.DataFrame]:
+ """
+ Split a combined ``GROUPING SETS`` result into one DataFrame per rollup
+ level, the inverse of {@link grouping_sets_clause}.
+
+ Each row of ``df`` carries a ``GROUPING()`` marker per groupby column (named
+ by ``grouping_marker_label``): ``0`` if the column is grouped at that row's
+ level, ``1`` if it was rolled up. A row belongs to a level iff its markers
+ are ``0`` exactly for that level's columns. Marker columns are dropped from
+ the returned frames so each looks like an ordinary per-level query result.
+
+ :param df: the combined query result, including marker columns
+ :param levels: the grouped-column list for each rollup level (same order as
+ passed to ``grouping_sets_clause``)
+ :param groupby_columns: every groupby column that has a marker
+ :return: one DataFrame per level, in ``levels`` order
+ """
+ markers = [grouping_marker_label(col) for col in groupby_columns]
+ results: list[pd.DataFrame] = []
+ for level in levels:
+ grouped = set(level)
+ mask = pd.Series(True, index=df.index)
Review Comment:
Added the type hint.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(table/pivot-table): correct non-additive totals/subtotals via DB rollup [POC for SIP-216] [superset]
Posted by "rusackas (via GitHub)" <gi...@apache.org>.
rusackas commented on code in PR #41184:
URL: https://github.com/apache/superset/pull/41184#discussion_r3565611254
##########
superset-frontend/plugins/plugin-chart-pivot-table/src/PivotTableChart.tsx:
##########
@@ -341,22 +296,45 @@ export default function PivotTableChart(props: PivotTableProps) {
const unpivotedData = useMemo(
() =>
- data.reduce(
- (acc: Record<string, any>[], record: Record<string, any>) => [
- ...acc,
- ...metricNames
+ // `data` is now one entry per rollup level. Tag every record with the
+ // row/column dimension labels of the level that produced it (mirroring
+ // the METRIC_KEY injection used for the full rows/cols below) so
+ // PivotData can slot each pre-computed value without re-aggregating.
+ // buildGroupbyCombinations already applied transposePivot, so the level's
+ // groupby is display-oriented and is not transposed again here.
+ data.flatMap((query: QueryData) => {
+ let levelRows = query.groupby.rows.map(getColumnLabel);
+ let levelCols = query.groupby.columns.map(getColumnLabel);
+ if (metricsLayout === MetricsLayoutEnum.ROWS) {
+ levelRows = combineMetric
+ ? [...levelRows, METRIC_KEY]
+ : [METRIC_KEY, ...levelRows];
+ } else {
+ levelCols = combineMetric
+ ? [...levelCols, METRIC_KEY]
+ : [METRIC_KEY, ...levelCols];
+ }
+ return query.data.flatMap((record: Record<string, any>) =>
+ metricNames
.map((name: string) => ({
...record,
[METRIC_KEY]: name,
value: record[name],
// Mark currency column for per-cell currency detection in aggregators
__currencyColumn: currencyCodeColumn,
+ // The level this record belongs to (used by PivotData placement).
+ // Namespaced with a `__` prefix (like `__metricKey` below) so it
+ // can't collide with a real dataset column named `rows`/`columns`.
+ __rows: levelRows,
+ __columns: levelCols,
+ // Identify the metric pseudo-dimension so PivotData can feed the
+ // metric-collapsed totals (the opposite "Total" axis + corner).
+ __metricKey: METRIC_KEY,
}))
- .filter(record => record.value !== null),
- ],
- [],
- ),
- [data, metricNames, currencyCodeColumn],
+ .filter(r => r.value !== null),
Review Comment:
Good catch, this could drop a whole row instead of showing a blank cell when a DB-computed total is null. Kept the record and taught the passthrough formatter to render `null` as an empty cell instead of the literal `null` string.
##########
superset/common/query_context_processor.py:
##########
@@ -252,9 +256,60 @@ def get_query_result(self, query_object: QueryObject) -> QueryResult:
This method delegates to the datasource's get_query_result method,
which handles query execution, normalization, time offsets, and
post-processing.
+
+ When the query requests rollup ``grouping_sets`` but the engine does not
+ support native ``GROUPING SETS``, fall back to one query per level and
+ concatenate the results with ``GROUPING()``-equivalent markers, so the
+ combined result matches the shape the native path produces (SIP.md,
+ phase 3b). Engines that support it run the single native query.
"""
+ if query_object.grouping_sets and not self._supports_grouping_sets():
+ return self._grouping_sets_fallback(query_object)
return self._qc_datasource.get_query_result(query_object)
+ def _supports_grouping_sets(self) -> bool:
+ engine_spec: BaseEngineSpec | None = getattr(
+ self._qc_datasource, "db_engine_spec", None
+ )
+ return bool(engine_spec and engine_spec.supports_grouping_sets)
+
+ def _grouping_sets_fallback(self, query_object: QueryObject) -> QueryResult:
+ """
+ Emulate a GROUPING SETS query on engines without native support: run one
+ query per rollup level and concatenate, tagging each level's rows with
+ the same per-column markers the native path emits.
+ """
+ levels: list[list[str]] = query_object.grouping_sets
+ # Use the same label derivation as the native path (physical column name
+ # or adhoc column label) so both column kinds are represented and each
+ # label maps back to its own column, in the same order as the source
+ # list.
+ all_labels = [get_column_name(col) for col in query_object.columns]
+ label_to_column = dict(zip(all_labels, query_object.columns, strict=True))
+
+ frames: list[pd.DataFrame] = []
+ result: QueryResult | None = None
+ for level in levels:
+ level_labels = set(level)
+ sub_query = copy.copy(query_object)
+ sub_query.grouping_sets = []
+ sub_query.columns = [
+ label_to_column[label] for label in all_labels if label in level_labels
+ ]
Review Comment:
Good catch, fixed by clearing `row_limit` on each fallback level. The native GROUPING SETS path skips it entirely (see the `use_grouping_sets` check in `models/helpers.py`), so this brings the fallback in line with it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(table/pivot-table): correct non-additive totals/subtotals via DB rollup [POC for SIP-216] [superset]
Posted by "rusackas (via GitHub)" <gi...@apache.org>.
rusackas commented on code in PR #41184:
URL: https://github.com/apache/superset/pull/41184#discussion_r3565629696
##########
superset-frontend/plugins/plugin-chart-pivot-table/src/plugin/buildQuery.ts:
##########
@@ -54,6 +59,39 @@ export default function buildQuery(formData: PivotTableQueryFormData) {
}
return col;
});
+}
+
+export default function buildQuery(formData: PivotTableQueryFormData) {
+ const { extra_form_data } = formData;
+ const time_grain_sqla =
+ extra_form_data?.time_grain_sqla || formData.time_grain_sqla;
Review Comment:
That is not right, `buildQueryContext` already populates `extras.time_grain_sqla` from `formData.time_grain_sqla` by default. The `time_grain_sqla` here is only used to tag the BASE_AXIS column, and there is already a passing test (`should not omit extras.time_grain_sqla from queryContext so dashboards apply them`) covering the override.
##########
superset-frontend/plugins/plugin-chart-pivot-table/test/react-pivottable/tableRenders.test.tsx:
##########
@@ -149,18 +202,83 @@ test('TableRenderer renders col totals row when colTotals is enabled', () => {
test('TableRenderer renders grand total when both totals are enabled', () => {
const props = buildDefaultProps({
+ data: TAGGED_COUNT_DATA,
+ vals: ['value'],
tableOptions: { rowTotals: true, colTotals: true },
});
renderWithTheme(<TableRenderer {...props} />);
- // The grand total cell should show "4" (total record count).
+ // The grand total cell shows the DB-computed "4" (total record count).
const grandTotalCells = screen
.getAllByRole('gridcell')
.filter(cell => cell.classList.contains('pvtGrandTotal'));
expect(grandTotalCells.length).toBe(1);
expect(grandTotalCells[0]).toHaveTextContent('4');
Review Comment:
`toHaveTextContent` does a substring match by default, so `4.00` satisfies `toHaveTextContent(4)`. Test passes as written.
##########
superset-frontend/plugins/plugin-chart-pivot-table/src/react-pivottable/utilities.ts:
##########
@@ -1088,76 +1117,108 @@ class PivotData {
}
processRecord(record: PivotRecord): void {
- // this code is called in a tight loop
+ // this code is called in a tight loop.
+ // Each record is tagged (in PivotTableChart) with `__rows`/`__columns`:
+ // the dimension labels of the rollup level that produced it. The database
+ // has already aggregated that level, so we place the value into exactly
+ // one slot and store it verbatim (no re-aggregation). A key shorter than
+ // the full dimension list identifies a subtotal level. Records without
+ // tags (e.g. direct unit-test construction) fall back to the full
+ // dimension lists, i.e. they are treated as leaf rows. The `__` prefix
+ // keeps these rollup tags from colliding with a real dataset column
+ // named `rows`/`columns`.
+ const recordRows = (record.__rows as unknown as string[]) ?? null;
+ const recordColumns = (record.__columns as unknown as string[]) ?? null;
+ const levelRows = recordRows ?? (this.props.rows as string[]);
+ const levelColumns = recordColumns ?? (this.props.cols as string[]);
+
const colKey: string[] = [];
const rowKey: string[] = [];
- (this.props.cols as string[]).forEach((col: string) => {
+ levelColumns.forEach((col: string) => {
colKey.push(col in record ? String(record[col]) : 'null');
});
- (this.props.rows as string[]).forEach((row: string) => {
+ levelRows.forEach((row: string) => {
rowKey.push(row in record ? String(record[row]) : 'null');
});
- this.allTotal.push(record);
-
- const rowStart = this.subtotals.rowEnabled ? 1 : Math.max(1, rowKey.length);
- const colStart = this.subtotals.colEnabled ? 1 : Math.max(1, colKey.length);
-
- let isRowSubtotal;
- let isColSubtotal;
- for (let ri = rowStart; ri <= rowKey.length; ri += 1) {
- isRowSubtotal = ri < rowKey.length;
- const fRowKey = rowKey.slice(0, ri);
- const flatRowKey = flatKey(fRowKey);
- if (!this.rowTotals[flatRowKey]) {
- this.rowKeys.push(fRowKey);
- this.rowTotals[flatRowKey] = this.getFormattedAggregator(
- record,
- rowKey,
- )(this, fRowKey, []);
+ const flatRowKey = flatKey(rowKey);
+ const flatColKey = flatKey(colKey);
+
+ const isColSubtotal = colKey.length < (this.props.cols as string[]).length;
+ const isRowSubtotal = rowKey.length < (this.props.rows as string[]).length;
+
+ // Register/create the column-total slot the first time this colKey is seen.
+ if (colKey.length > 0 && !this.colTotals[flatColKey]) {
+ if (!isColSubtotal || this.subtotals.colEnabled) {
+ this.colKeys.push(colKey);
}
- this.rowTotals[flatRowKey].push(record);
- this.rowTotals[flatRowKey].isSubtotal = isRowSubtotal;
+ this.colTotals[flatColKey] = this.getFormattedAggregator(record, colKey)(
+ this,
+ [],
+ colKey,
+ );
}
-
- for (let ci = colStart; ci <= colKey.length; ci += 1) {
- isColSubtotal = ci < colKey.length;
- const fColKey = colKey.slice(0, ci);
- const flatColKey = flatKey(fColKey);
- if (!this.colTotals[flatColKey]) {
- this.colKeys.push(fColKey);
- this.colTotals[flatColKey] = this.getFormattedAggregator(
- record,
+ // Register/create the row-total slot the first time this rowKey is seen.
+ if (rowKey.length > 0 && !this.rowTotals[flatRowKey]) {
+ if (!isRowSubtotal || this.subtotals.rowEnabled) {
+ this.rowKeys.push(rowKey);
+ }
+ this.rowTotals[flatRowKey] = this.getFormattedAggregator(record, rowKey)(
+ this,
+ rowKey,
+ [],
+ );
+ }
+ // Create the body-cell slot.
+ if (rowKey.length > 0 && colKey.length > 0) {
+ if (!this.tree[flatRowKey]) {
+ this.tree[flatRowKey] = {};
+ }
+ if (!this.tree[flatRowKey][flatColKey]) {
+ this.tree[flatRowKey][flatColKey] = this.getFormattedAggregator(record)(
+ this,
+ rowKey,
colKey,
- )(this, [], fColKey);
+ );
}
+ }
+
+ // Place the value in exactly one slot, determined by the level.
+ if (rowKey.length === 0 && colKey.length === 0) {
+ this.allTotal.push(record);
+ } else if (rowKey.length === 0) {
this.colTotals[flatColKey].push(record);
this.colTotals[flatColKey].isSubtotal = isColSubtotal;
+ } else if (colKey.length === 0) {
+ this.rowTotals[flatRowKey].push(record);
+ this.rowTotals[flatRowKey].isSubtotal = isRowSubtotal;
+ } else {
+ this.tree[flatRowKey][flatColKey].push(record);
+ this.tree[flatRowKey][flatColKey].isRowSubtotal = isRowSubtotal;
+ this.tree[flatRowKey][flatColKey].isColSubtotal = isColSubtotal;
+ this.tree[flatRowKey][flatColKey].isSubtotal =
+ isRowSubtotal || isColSubtotal;
}
- // And now fill in for all the sub-cells.
- for (let ri = rowStart; ri <= rowKey.length; ri += 1) {
- isRowSubtotal = ri < rowKey.length;
- const fRowKey = rowKey.slice(0, ri);
- const flatRowKey = flatKey(fRowKey);
- if (!this.tree[flatRowKey]) {
- this.tree[flatRowKey] = {};
+ // Metric-collapse totals. The metric is a pseudo-dimension always present on
+ // one axis, so no rollup level produces an empty key on that axis -- which
+ // would leave the opposite "Total" axis and the grand-total corner empty.
+ // When a record's axis holds only the metric (no real dims there), its value
+ // is also the collapsed total for that axis, so mirror it into rowTotals /
+ // colTotals / allTotal. (For a single metric this equals the metric column;
+ // for multiple metrics it is the last metric -- a cross-metric total is not
+ // well defined and is left as future work.)
+ const metricKey = record.__metricKey as unknown as string | undefined;
+ if (metricKey) {
+ const realColCount = levelColumns.filter(c => c !== metricKey).length;
+ const realRowCount = levelRows.filter(r => r !== metricKey).length;
+ if (levelColumns.includes(metricKey) && realColCount === 0) {
+ if (rowKey.length === 0) this.allTotal.push(record);
+ else this.rowTotals[flatRowKey]?.push(record);
}
- for (let ci = colStart; ci <= colKey.length; ci += 1) {
- isColSubtotal = ci < colKey.length;
- const fColKey = colKey.slice(0, ci);
- const flatColKey = flatKey(fColKey);
- if (!this.tree[flatRowKey][flatColKey]) {
- this.tree[flatRowKey][flatColKey] = this.getFormattedAggregator(
- record,
- )(this, fRowKey, fColKey);
- }
- this.tree[flatRowKey][flatColKey].push(record);
-
- this.tree[flatRowKey][flatColKey].isRowSubtotal = isRowSubtotal;
- this.tree[flatRowKey][flatColKey].isColSubtotal = isColSubtotal;
- this.tree[flatRowKey][flatColKey].isSubtotal =
- isRowSubtotal || isColSubtotal;
+ if (levelRows.includes(metricKey) && realRowCount === 0) {
+ if (colKey.length === 0) this.allTotal.push(record);
+ else this.colTotals[flatColKey]?.push(record);
}
Review Comment:
Different toggle: `subtotals.colEnabled`/`rowEnabled` gate intermediate subtotal levels, not the fully-collapsed totals level this mirrors into. When `colTotals`/`rowTotals` are off, `buildGroupbyCombinations` never even requests that level, so `processRecord` never sees these records to begin with.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(table/pivot-table): correct non-additive totals/subtotals via DB rollup [POC for SIP-216] [superset]
Posted by "bito-code-review[bot] (via GitHub)" <gi...@apache.org>.
bito-code-review[bot] commented on code in PR #41184:
URL: https://github.com/apache/superset/pull/41184#discussion_r3565630723
##########
superset-frontend/plugins/plugin-chart-pivot-table/src/react-pivottable/TableRenderers.tsx:
##########
@@ -94,7 +94,6 @@ interface SubtotalOptions {
interface TableRendererProps {
cols: string[];
rows: string[];
- aggregatorName: string;
tableOptions?: TableOptions;
Review Comment:
<!-- Bito Reply -->
The suggestion to reintroduce `aggregatorName` is not necessary in this context. As you noted, the removal of the per-metric aggregation function control means this prop is no longer used for client-side selection, and keeping it would introduce unused code.
**superset-frontend/plugins/plugin-chart-pivot-table/src/react-pivottable/TableRenderers.tsx**
```
interface TableRendererProps {
cols: string[];
rows: string[];
tableOptions?: TableOptions;
```
##########
superset-frontend/plugins/plugin-chart-pivot-table/src/react-pivottable/TableRenderers.tsx:
##########
@@ -1099,9 +1096,7 @@ export function TableRenderer(props: TableRendererProps) {
true,
)}
>
- {t('Total (%(aggregatorName)s)', {
- aggregatorName: t(aggregatorName),
- })}
+ {t('Total')}
Review Comment:
<!-- Bito Reply -->
The suggestion to restore the translation pattern is not necessary in this context. As noted, the aggregation function control has been removed, making the simplified 'Total' label appropriate for the current implementation.
##########
superset-frontend/plugins/plugin-chart-pivot-table/src/react-pivottable/TableRenderers.tsx:
##########
@@ -1448,9 +1437,7 @@ export function TableRenderer(props: TableRendererProps) {
true,
)}
>
- {t('Total (%(aggregatorName)s)', {
- aggregatorName: t(aggregatorName),
- })}
+ {t('Total')}
Review Comment:
<!-- Bito Reply -->
The suggestion to restore the `aggregatorName` in the total column header is a review comment, not a mandatory requirement. If the `aggregatorName` control has been intentionally removed as part of the changes described in `UPDATING.md`, then the suggestion is not applicable to the current state of the code. You may choose to ignore this suggestion if it conflicts with the intended design changes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(table/pivot-table): correct non-additive totals/subtotals via DB rollup [POC for SIP-216] [superset]
Posted by "rusackas (via GitHub)" <gi...@apache.org>.
rusackas commented on code in PR #41184:
URL: https://github.com/apache/superset/pull/41184#discussion_r3565630469
##########
superset-frontend/plugins/plugin-chart-pivot-table/src/react-pivottable/TableRenderers.tsx:
##########
@@ -94,7 +94,6 @@ interface SubtotalOptions {
interface TableRendererProps {
cols: string[];
rows: string[];
- aggregatorName: string;
tableOptions?: TableOptions;
Review Comment:
This is deliberate, not a bypass: the per-metric "Aggregation function" control is removed by this PR since totals are now DB-computed per metric definition rather than a single client-chosen function, see UPDATING.md. `Total (%(aggregatorName)s)` no longer makes sense without it.
##########
superset-frontend/plugins/plugin-chart-pivot-table/src/react-pivottable/TableRenderers.tsx:
##########
@@ -1099,9 +1096,7 @@ export function TableRenderer(props: TableRendererProps) {
true,
)}
>
- {t('Total (%(aggregatorName)s)', {
- aggregatorName: t(aggregatorName),
- })}
+ {t('Total')}
Review Comment:
Same as the other `aggregatorName` threads on this file, this is intentional per UPDATING.md since the Aggregation function control is gone.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org