You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@superset.apache.org by "mistercrunch (via GitHub)" <gi...@apache.org> on 2026/02/11 02:35:42 UTC
[PR] fix(viz): flatten MultiIndex columns in Time-Series Table for multiple Group By [superset]
mistercrunch opened a new pull request, #37869:
URL: https://github.com/apache/superset/pull/37869
## Summary
Fixes #12066 — a 5-year-old bug where using multiple "Group By" columns in a **Time-Series Table** chart causes:
```
TypeError: keys must be str, int, float, bool or None, not tuple
```
### Root Cause
When multiple `groupby` columns are specified, `df.pivot_table()` produces a `pd.MultiIndex` on columns (tuples like `("category_a", "region_1")`). Calling `pt.to_dict(orient="index")` then fails because JSON cannot serialize tuple keys.
### Fix
After `pivot_table()` and sorting, flatten any `MultiIndex` columns to comma-separated strings before serialization:
```python
if isinstance(pt.columns, pd.MultiIndex):
pt.columns = [", ".join(str(s) for s in col) for col in pt.columns]
```
This is a **2-line change** in `superset/viz.py` (`TimeTableViz.get_data()`). The fix is compatible with the frontend, which already expects string column keys.
### Test Plan
- [x] Added `test_get_data_multiple_group_by` integration test that exercises the multiple-groupby path and asserts flattened string keys in both `records` and `columns`
- [x] Existing `TestTimeSeriesTableViz` tests continue to pass (single groupby and no-groupby paths unchanged)
### Confirmed Affected Versions
Reported across 0.26, 0.35, 1.1, 1.3, 1.4, 2.0.1, 2.1.3, 3.0.1, 4.0.1, and 4.1.1.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] fix(viz): flatten MultiIndex columns in Time-Series Table for multiple Group By [superset]
Posted by "rusackas (via GitHub)" <gi...@apache.org>.
rusackas merged PR #37869:
URL: https://github.com/apache/superset/pull/37869
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] fix(viz): flatten MultiIndex columns in Time-Series Table for multiple Group By [superset]
Posted by "codecov[bot] (via GitHub)" <gi...@apache.org>.
codecov[bot] commented on PR #37869:
URL: https://github.com/apache/superset/pull/37869#issuecomment-4318462526
## [Codecov](https://app.codecov.io/gh/apache/superset/pull/37869?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:x: Patch coverage is `33.33333%` with `2 lines` in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 55.42%. Comparing base ([`f255f63`](https://app.codecov.io/gh/apache/superset/commit/f255f63953cad337a6af74a87a8130d3695236eb?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`6cd7060`](https://app.codecov.io/gh/apache/superset/commit/6cd70602e6173afdc0bb525159eef31f931ceb29?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
| [Files with missing lines](https://app.codecov.io/gh/apache/superset/pull/37869?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [superset/viz.py](https://app.codecov.io/gh/apache/superset/pull/37869?src=pr&el=tree&filepath=superset%2Fviz.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-c3VwZXJzZXQvdml6LnB5) | 33.33% | [2 Missing :warning: ](https://app.codecov.io/gh/apache/superset/pull/37869?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
> :exclamation: There is a different number of reports uploaded between BASE (f255f63) and HEAD (6cd7060). Click for more details.
>
> <details><summary>HEAD has 54 uploads less than BASE</summary>
>
>| Flag | BASE (f255f63) | HEAD (6cd7060) |
>|------|------|------|
>|python|50|1|
>|presto|6|1|
></details>
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #37869 +/- ##
==========================================
- Coverage 64.47% 55.42% -9.06%
==========================================
Files 2566 2566
Lines 133925 133928 +3
Branches 31095 31096 +1
==========================================
- Hits 86349 74227 -12122
- Misses 46081 59046 +12965
+ Partials 1495 655 -840
```
| [Flag](https://app.codecov.io/gh/apache/superset/pull/37869/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [hive](https://app.codecov.io/gh/apache/superset/pull/37869/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [mysql](https://app.codecov.io/gh/apache/superset/pull/37869/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [postgres](https://app.codecov.io/gh/apache/superset/pull/37869/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [presto](https://app.codecov.io/gh/apache/superset/pull/37869/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `41.54% <33.33%> (-0.01%)` | :arrow_down: |
| [python](https://app.codecov.io/gh/apache/superset/pull/37869/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `41.54% <33.33%> (-20.33%)` | :arrow_down: |
| [sqlite](https://app.codecov.io/gh/apache/superset/pull/37869/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [unit](https://app.codecov.io/gh/apache/superset/pull/37869/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
</details>
[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/apache/superset/pull/37869?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
- :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] fix(viz): flatten MultiIndex columns in Time-Series Table for multiple Group By [superset]
Posted by "bito-code-review[bot] (via GitHub)" <gi...@apache.org>.
bito-code-review[bot] commented on PR #37869:
URL: https://github.com/apache/superset/pull/37869#issuecomment-3881777523
AI Code Review is in progress (usually takes 3 to 15 minutes unless it's a very large PR).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] fix(viz): flatten MultiIndex columns in Time-Series Table for multiple Group By [superset]
Posted by "rusackas (via GitHub)" <gi...@apache.org>.
rusackas commented on PR #37869:
URL: https://github.com/apache/superset/pull/37869#issuecomment-4307171166
Code looks good to me — the MultiIndex flatten is targeted, reuses the existing `FLAT_COLUMN_SEPARATOR`/`escape_separator` helpers (so the Copilot threads are effectively addressed; safe to mark resolved), and the integration test covers the multi-groupby path. The only thing holding this back is the rebase you mentioned — #37871 merged back on Feb 11, so a rebase should clear the unrelated `test_extension_config_full` failures and get CI green. Happy to approve once that's done.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] fix(viz): flatten MultiIndex columns in Time-Series Table for multiple Group By [superset]
Posted by "mistercrunch (via GitHub)" <gi...@apache.org>.
mistercrunch commented on code in PR #37869:
URL: https://github.com/apache/superset/pull/37869#discussion_r2791188202
##########
superset/viz.py:
##########
@@ -747,6 +747,8 @@ def get_data(self, df: pd.DataFrame) -> VizData:
pt = df.pivot_table(index=DTTM_ALIAS, columns=columns, values=values)
pt.index = pt.index.map(str)
pt = pt.sort_index()
+ if isinstance(pt.columns, pd.MultiIndex):
+ pt.columns = [", ".join(str(s) for s in col) for col in pt.columns]
Review Comment:
Good catch — updated to use `escape_separator` and `FLAT_COLUMN_SEPARATOR` from `superset.utils.pandas_postprocessing.utils` for consistency and to handle commas in group-by values.
##########
tests/integration_tests/viz_tests.py:
##########
@@ -626,6 +626,39 @@ def test_get_data_group_by(self):
}
assert expected == data["records"]
+ def test_get_data_multiple_group_by(self):
+ form_data = {"metrics": ["sum__A"], "groupby": ["groupby1", "groupby2"]}
+ datasource = self.get_datasource_mock()
+ raw = {}
+ t1 = pd.Timestamp("2000")
+ t2 = pd.Timestamp("2002")
+ raw[DTTM_ALIAS] = [t1, t1, t1, t1, t2, t2, t2, t2]
+ raw["sum__A"] = [15, 20, 25, 30, 35, 40, 45, 50]
+ raw["groupby1"] = ["a1", "a2", "a1", "a2", "a1", "a2", "a1", "a2"]
+ raw["groupby2"] = ["b1", "b1", "b2", "b2", "b1", "b1", "b2", "b2"]
+ df = pd.DataFrame(raw)
+ test_viz = viz.TimeTableViz(datasource, form_data)
+ data = test_viz.get_data(df)
+ # Columns should be flattened strings, not tuples
+ assert {"a1, b1", "a1, b2", "a2, b1", "a2, b2"} == set(data["columns"])
+ time_format = "%Y-%m-%d %H:%M:%S"
+ expected = {
+ t1.strftime(time_format): {
+ "a1, b1": 15,
+ "a1, b2": 25,
+ "a2, b1": 20,
+ "a2, b2": 30,
+ },
+ t2.strftime(time_format): {
+ "a1, b1": 35,
+ "a1, b2": 45,
+ "a2, b1": 40,
+ "a2, b2": 50,
Review Comment:
Updated — the test now uses `FLAT_COLUMN_SEPARATOR` via a `sep` variable throughout.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] fix(viz): flatten MultiIndex columns in Time-Series Table for multiple Group By [superset]
Posted by "bito-code-review[bot] (via GitHub)" <gi...@apache.org>.
bito-code-review[bot] commented on PR #37869:
URL: https://github.com/apache/superset/pull/37869#issuecomment-3881980608
AI Code Review is in progress (usually takes 3 to 15 minutes unless it's a very large PR).
<details>
<summary>
Bito Usage Guide
</summary>
**Commands**
Type the following command in the pull request comment and save the comment.
- `/review` - Manually triggers a full AI review.
- `/pause` - Pauses automatic reviews on this pull request.
- `/resume` - Resumes automatic reviews.
- `/resolve` - Marks all Bito-posted review comments as resolved.
- `/abort` - Cancels all in-progress reviews.
Refer to the <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/available-commands" target="_blank" rel="noopener noreferrer">documentation</a> for additional commands.
**Configuration**
This repository uses `Superset` You can customize the agent settings <a href="https://alpha.bito.ai/home/ai-agents/code-review-agent" target="_blank" rel="noopener noreferrer">here</a> or contact your Bito workspace admin at evan@preset.io.
**Documentation & Help**
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/getting-started/install-run-using-bito-cloud/create-or-customize-an-agent-instance" target="_blank" rel="noopener noreferrer">Customize agent settings</a>
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/implementing-custom-code-review-rules" target="_blank" rel="noopener noreferrer">Review rules</a>
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent" target="_blank" rel="noopener noreferrer">General documentation</a>
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/faqs" target="_blank" rel="noopener noreferrer">FAQ</a>
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] fix(viz): flatten MultiIndex columns in Time-Series Table for multiple Group By [superset]
Posted by "mistercrunch (via GitHub)" <gi...@apache.org>.
mistercrunch commented on PR #37869:
URL: https://github.com/apache/superset/pull/37869#issuecomment-3882224206
The 3 failing `unit-tests` jobs (current/next/previous) are all the same pre-existing failure in `tests/unit_tests/extensions/test_types.py::test_extension_config_full` — introduced by #37856 which changed the extension contribution schema without updating this test.
This is already being fixed in #37871. Unrelated to the changes in this PR (4,662 tests pass across all 3 Python versions).
Will rebase once #37871 merges to get clean CI.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] fix(viz): flatten MultiIndex columns in Time-Series Table for multiple Group By [superset]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #37869:
URL: https://github.com/apache/superset/pull/37869#discussion_r2791164313
##########
superset/viz.py:
##########
@@ -747,6 +747,8 @@ def get_data(self, df: pd.DataFrame) -> VizData:
pt = df.pivot_table(index=DTTM_ALIAS, columns=columns, values=values)
pt.index = pt.index.map(str)
pt = pt.sort_index()
+ if isinstance(pt.columns, pd.MultiIndex):
+ pt.columns = [", ".join(str(s) for s in col) for col in pt.columns]
Review Comment:
Flattening MultiIndex columns via `', '.join(...)` can create ambiguous/duplicate column names when any group-by value contains a comma, which can silently overwrite values when `pt.to_dict(orient='index')` is built. The codebase already has MultiIndex-flattening helpers/constants (`superset.utils.pandas_postprocessing.flatten.flatten` and `escape_separator`/`FLAT_COLUMN_SEPARATOR`) that escape separators to avoid collisions; consider reusing those here (or at least escaping the separator) for consistent and safe serialization.
##########
tests/integration_tests/viz_tests.py:
##########
@@ -626,6 +626,39 @@ def test_get_data_group_by(self):
}
assert expected == data["records"]
+ def test_get_data_multiple_group_by(self):
+ form_data = {"metrics": ["sum__A"], "groupby": ["groupby1", "groupby2"]}
+ datasource = self.get_datasource_mock()
+ raw = {}
+ t1 = pd.Timestamp("2000")
+ t2 = pd.Timestamp("2002")
+ raw[DTTM_ALIAS] = [t1, t1, t1, t1, t2, t2, t2, t2]
+ raw["sum__A"] = [15, 20, 25, 30, 35, 40, 45, 50]
+ raw["groupby1"] = ["a1", "a2", "a1", "a2", "a1", "a2", "a1", "a2"]
+ raw["groupby2"] = ["b1", "b1", "b2", "b2", "b1", "b1", "b2", "b2"]
+ df = pd.DataFrame(raw)
+ test_viz = viz.TimeTableViz(datasource, form_data)
+ data = test_viz.get_data(df)
+ # Columns should be flattened strings, not tuples
+ assert {"a1, b1", "a1, b2", "a2, b1", "a2, b2"} == set(data["columns"])
+ time_format = "%Y-%m-%d %H:%M:%S"
+ expected = {
+ t1.strftime(time_format): {
+ "a1, b1": 15,
+ "a1, b2": 25,
+ "a2, b1": 20,
+ "a2, b2": 30,
+ },
+ t2.strftime(time_format): {
+ "a1, b1": 35,
+ "a1, b2": 45,
+ "a2, b1": 40,
+ "a2, b2": 50,
Review Comment:
This test hard-codes the `", "` separator in expected column names. Since the backend already defines `FLAT_COLUMN_SEPARATOR = ", "` for MultiIndex flattening, consider referencing that constant (or building expectations from it) to avoid the test becoming stale if the separator ever changes.
```suggestion
sep = viz.FLAT_COLUMN_SEPARATOR
assert {f"a1{sep}b1", f"a1{sep}b2", f"a2{sep}b1", f"a2{sep}b2"} == set(
data["columns"]
)
time_format = "%Y-%m-%d %H:%M:%S"
expected = {
t1.strftime(time_format): {
f"a1{sep}b1": 15,
f"a1{sep}b2": 25,
f"a2{sep}b1": 20,
f"a2{sep}b2": 30,
},
t2.strftime(time_format): {
f"a1{sep}b1": 35,
f"a1{sep}b2": 45,
f"a2{sep}b1": 40,
f"a2{sep}b2": 50,
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org